Curl loader is efficient tool implemented in c language for application generation and behaviour of thousands of virtual HTTP/HTTPS and FTP/FTPS clients.
This tool provide different options for generation and behaviour of virtual client .Some of them described here like
1)User can create every virtual client with unique or same IP address
2) Generation of clients with specified increase of clients after every cycle
3) Its allows clients to use methods GET ,POST and PUT.
4)HTTP user authentication login with POST or GET+POST methods.
5)For user authentication ,curl loader allows to provide unique user name and password or same username and password or credentials from text file.
6)HTTP multipart form data using POST
7)Customization of client request HTTP/FTP headers
8)Set up Proxy Authentication method and provide credentials
and many others
Curl loader can be download from link and you can read its detailed documentation from this page
Building
Download curl loader from above link and extract into a folder.
Make sure you install SSL development library
While installation i got errors related to " rtmp ", i tried some ways to get rid of it .
I tried adding the option
finally i tried installing on other system, i didn't got any errors during installation.
USING CURL-LOADER
As shown above figure , I started working with given 10K.conf to get the basic working of curl loader .
Change the path in terminal to folder where the curl loader is installed and use command
#curl-loader -f ./conf-examples/bulk.conf
Working with thousands of clients require to change the increase limit of descriptors
by login as root
$ su root
enter password
then below command
#ulimit -n 100 000
Some of terms need to be understood for using the configuration file:
BATCH_NAME requires a string value; it is used to name the batch.
CLIENTS_NUM_MAX is the maximum number of clients to be used for this load.
CLIENTS_NUM_START is the number of clients to start the load with.
CLIENTS_RAMPUP_INC number of clients to be added to the load in the auto mode every second till CLIENTS_NUM_MAX number will be reached.
INTERFACE - name of the loading network interface, like eth0. The network interface names can be checked by running ifconfig in terminal.
NETMASK - netmask for the loading IP-addresses of virtual clients .
IP-addresses from IP_ADDR_MIN up to IP_ADDR_MAX are the interval of addresses to be used for loading clients. The interval of addresses should be large enough to supply a dedicated IP for each client.
Another mode that can be used is "single IP address" for all clients. When IP_ADDR_MIN and IP_ADDR_MAX have the same valid IP-address, it will be used for all clients.
CYCLES_NUM is the number of cycles to be performed. Any positive value is valid here as well as (-1), which means cycle indefinitely. Loading can be normally stopped at any moment by pressing Cntl-C
URLS_NUM is the number of urls to be used in URLs Section.
URL SECTION
URL: The tag should be either a valid url or an empty string,which is allowed only when URL_USE_CURRENT=1.This tag is required when needed to post data to URL previously fetched by GET method.
URL_SHORT_NAME is optional string name appears at console
REQUEST_TYPE is use the methods POST ,GET and PUT
UPLOAD_FILE - a filename, including path, of a file to be used for uploading.
HEADER - is assisting to customize/add/over-write HTTP/FTP headers
USERNAME and PASSWORD are the tags to provide credentials for login operations.
FORM_USAGE_TYPE governs the login process like SINGLE_USER ,same username and password for all clients,"UNIQUE_USERS_SAME_PASSWORD"
for unique user name and password every client,"RECORDS_FROM_FILE" credentials fetched from the text file.
FORM_STRING template form to be used for POST-ing credentials
eg for SINGLE_USER "user=%s&secret=%s" and all clients will have the same POST credentials with the string looking like "user=robert&secret=stam".
FORM_RECORDS_FILE specifies path to the file with credentials
PROXY_AUTH_METHOD to be configured from the "BASIC", "DIGEST", "GSS_NEGOTIATE", "NTLM" or "ANY.
PROXY_AUTH_CREDENTIALS to be provided in the form "user:password".
for more detailed information regarding the curl loader follow this link.
I started working on make change in the source code of curl loader for fetching the headers from the text file instead of providing the headers in configuration file(.conf) for clients.
By fetching from the file i would be able to provide batch of headers for every clients
I had doubts what is HTTP headers and what is purpose.
Headers carry information about the client browser, the requested page, the server and more.
When i access the url, browser sends request headers (in red colour in screen shot )to server.Server responds by sending http respond headers and requested content.(headers in blue colour)
Note: Used wireshark tool for the analysing the interaction between client and server.
Http request structure
In first line"Request line" we find method name "GET" along with path and HTTP protocol.
Method informs about kind of request made by client.
Path is part of url after the host name.
eg http://www.papon.in/music.php
www.papon.in :host name
/music.php : path
Protocol here is HTTP 1.1
Others lines contain Headers
Accept : can be used to specify certain media types which are acceptable for the response
User agent: the details of browser and version
Http response structure
The first line "Status line" contains protocol and status code.
Status code means
Etag :for caching purposes
Content-Length:When content is going to be transmitted to the browser, the server can indicate the size of it (in bytes) using this header.
Below http response structure ,server sends content of requested data.
For more information on HTTP Headers ,check this "HTTP Headers for Dummies" .
Since i was trying to fetch the headers from text file and upload it . So for this purpose i found the concerned C file which would help me making these changes
1-parse-config.c
2-url.h
3-loader.c
Parse-config.c
In this file, i made changes in function "static int header_parser ( )"
1) "HEADER " tag in configuration file should contain either valid header value or check for presence of sub string "PATH>" which means headers to be fetched from the given path like "PATH>/home/user/documemts/hdrs.txt" .
2) Next, allocate the memory for storing the headers for clients ,so defined pointer to structure and each structure stores strings for a client.So for maximum number of clients ,defined array of structure which could store headers for every client.
3)Then task was to extract the strings for a text file and used separator ">" for separating the data for every client.
Below image shows the format of input text file for headers for 3 clients
For this input file, function read the strings from the file ,creates the array of three structure. Each structure stores strings like "Expect: 100-continue"and " Accept: text/plain" for first client for specific url.
While compiling code for this purpose , most often error i would get was "segmentation fault" which simply occurs accessing the pointer or data structure which has not been allocated memory.
More information on segmentation fault , click here.
For debugging ,may use GDB.
url.h
The data structure for storing the information provided by user through configuration file (.conf extension )in curl loader is stored by defining the structure "struct batch_context" in "batch.h" file which further is linked to number of "struct client_context" for client specific information defined in "client.h".Each client can access different urls so for each url , a defined structure "struct url_context " to store information defined in "url.h" file.So declaration of variables for structure for storing the headers for each url should be defined in "url.h".
loader.c
Final step was to upload the stored data for a client .For this purpose ,curl loader utilize the "libcurl ".The first task of declaration of pointer to linked list for posting like " struct curl_slist *list " in url.h file and set the list to null before using for next client.Then to access current client index and access its data from array of structure.Then append all strings in linked list using
curl_easy_setopt(CURL *handle, CURLoption option, parameter);
to post it .
finally after making all these changes in file, run make command to check the errors in code
and then make install.
TESTING
I made similar kind of file as shown above with maximum number of 3 client.Each client provided with different headers.So user has made 3 requests each some specific headers.
To check whether curl loader was able to upload the headers i used wireshark tool for analysing.
for installing wireshark
$sudo apt-get install wireshark
then for running
$ sudo wireshark
then dialog boxes will appear showing error
Just click Ok and use wireshark
In wireshark, it capture three requests which shows i am able to send 3 requests .
now check the request header(Expect: 100-continue, Accept-Encoding: gzip, deflat) for fist request and below screen shot verify that headers were posted for the request.
Thanks for reading.
Please leave your valuable comments.
This tool provide different options for generation and behaviour of virtual client .Some of them described here like
1)User can create every virtual client with unique or same IP address
2) Generation of clients with specified increase of clients after every cycle
3) Its allows clients to use methods GET ,POST and PUT.
4)HTTP user authentication login with POST or GET+POST methods.
5)For user authentication ,curl loader allows to provide unique user name and password or same username and password or credentials from text file.
6)HTTP multipart form data using POST
7)Customization of client request HTTP/FTP headers
8)Set up Proxy Authentication method and provide credentials
and many others
Curl loader can be download from link and you can read its detailed documentation from this page
Building
Download curl loader from above link and extract into a folder.
Make sure you install SSL development library
sudo apt-get install libssl-devWhile installation i got errors related to " rtmp ", i tried some ways to get rid of it .
I tried adding the option
--without-librtmp in Makefile, but i couldn't compile. Then adding install " rtmpdump" again it didn't work for me.finally i tried installing on other system, i didn't got any errors during installation.
USING CURL-LOADER
As shown above figure , I started working with given 10K.conf to get the basic working of curl loader .
Change the path in terminal to folder where the curl loader is installed and use command
#curl-loader -f ./conf-examples/bulk.conf
Working with thousands of clients require to change the increase limit of descriptors
by login as root
$ su root
enter password
then below command
#ulimit -n 100 000
Some of terms need to be understood for using the configuration file:
BATCH_NAME requires a string value; it is used to name the batch.
CLIENTS_NUM_MAX is the maximum number of clients to be used for this load.
CLIENTS_NUM_START is the number of clients to start the load with.
CLIENTS_RAMPUP_INC number of clients to be added to the load in the auto mode every second till CLIENTS_NUM_MAX number will be reached.
INTERFACE - name of the loading network interface, like eth0. The network interface names can be checked by running ifconfig in terminal.
NETMASK - netmask for the loading IP-addresses of virtual clients .
IP-addresses from IP_ADDR_MIN up to IP_ADDR_MAX are the interval of addresses to be used for loading clients. The interval of addresses should be large enough to supply a dedicated IP for each client.
Another mode that can be used is "single IP address" for all clients. When IP_ADDR_MIN and IP_ADDR_MAX have the same valid IP-address, it will be used for all clients.
CYCLES_NUM is the number of cycles to be performed. Any positive value is valid here as well as (-1), which means cycle indefinitely. Loading can be normally stopped at any moment by pressing Cntl-C
URLS_NUM is the number of urls to be used in URLs Section.
URL SECTION
URL: The tag should be either a valid url or an empty string,which is allowed only when URL_USE_CURRENT=1.This tag is required when needed to post data to URL previously fetched by GET method.
URL_SHORT_NAME is optional string name appears at console
REQUEST_TYPE is use the methods POST ,GET and PUT
UPLOAD_FILE - a filename, including path, of a file to be used for uploading.
HEADER - is assisting to customize/add/over-write HTTP/FTP headers
USERNAME and PASSWORD are the tags to provide credentials for login operations.
FORM_USAGE_TYPE governs the login process like SINGLE_USER ,same username and password for all clients,"UNIQUE_USERS_SAME_PASSWORD"
for unique user name and password every client,"RECORDS_FROM_FILE" credentials fetched from the text file.
FORM_STRING template form to be used for POST-ing credentials
eg for SINGLE_USER "user=%s&secret=%s" and all clients will have the same POST credentials with the string looking like "user=robert&secret=stam".
FORM_RECORDS_FILE specifies path to the file with credentials
PROXY_AUTH_METHOD to be configured from the "BASIC", "DIGEST", "GSS_NEGOTIATE", "NTLM" or "ANY.
PROXY_AUTH_CREDENTIALS to be provided in the form "user:password".
for more detailed information regarding the curl loader follow this link.
I started working on make change in the source code of curl loader for fetching the headers from the text file instead of providing the headers in configuration file(.conf) for clients.
By fetching from the file i would be able to provide batch of headers for every clients
I had doubts what is HTTP headers and what is purpose.
Headers carry information about the client browser, the requested page, the server and more.
When i access the url, browser sends request headers (in red colour in screen shot )to server.Server responds by sending http respond headers and requested content.(headers in blue colour)
Http request structure
In first line"Request line" we find method name "GET" along with path and HTTP protocol.
Method informs about kind of request made by client.
Path is part of url after the host name.
eg http://www.papon.in/music.php
www.papon.in :host name
/music.php : path
Protocol here is HTTP 1.1
Others lines contain Headers
Accept : can be used to specify certain media types which are acceptable for the response
User agent: the details of browser and version
Http response structure
The first line "Status line" contains protocol and status code.
Status code means
- 200's are used for successful requests.
- 300's are for redirections.
- 400's are used if there was a problem with the request.
- 500's are used if there was a problem with the server.
Etag :for caching purposes
Content-Length:When content is going to be transmitted to the browser, the server can indicate the size of it (in bytes) using this header.
Below http response structure ,server sends content of requested data.
For more information on HTTP Headers ,check this "HTTP Headers for Dummies" .
Since i was trying to fetch the headers from text file and upload it . So for this purpose i found the concerned C file which would help me making these changes
1-parse-config.c
2-url.h
3-loader.c
Parse-config.c
In this file, i made changes in function "static int header_parser ( )"
1) "HEADER " tag in configuration file should contain either valid header value or check for presence of sub string "PATH>" which means headers to be fetched from the given path like "PATH>/home/user/documemts/hdrs.txt" .
2) Next, allocate the memory for storing the headers for clients ,so defined pointer to structure and each structure stores strings for a client.So for maximum number of clients ,defined array of structure which could store headers for every client.
3)Then task was to extract the strings for a text file and used separator ">" for separating the data for every client.
Below image shows the format of input text file for headers for 3 clients
For this input file, function read the strings from the file ,creates the array of three structure. Each structure stores strings like "Expect: 100-continue"and " Accept: text/plain" for first client for specific url.
While compiling code for this purpose , most often error i would get was "segmentation fault" which simply occurs accessing the pointer or data structure which has not been allocated memory.
More information on segmentation fault , click here.
For debugging ,may use GDB.
url.h
The data structure for storing the information provided by user through configuration file (.conf extension )in curl loader is stored by defining the structure "struct batch_context" in "batch.h" file which further is linked to number of "struct client_context" for client specific information defined in "client.h".Each client can access different urls so for each url , a defined structure "struct url_context " to store information defined in "url.h" file.So declaration of variables for structure for storing the headers for each url should be defined in "url.h".
loader.c
Final step was to upload the stored data for a client .For this purpose ,curl loader utilize the "libcurl ".The first task of declaration of pointer to linked list for posting like " struct curl_slist *list " in url.h file and set the list to null before using for next client.Then to access current client index and access its data from array of structure.Then append all strings in linked list using
struct curl_slist *curl_slist_append(struct curl_slist * list, const char * string );
and use curl_easy_setopt(CURL *handle, CURLoption option, parameter);
to post it .
finally after making all these changes in file, run make command to check the errors in code
and then make install.
TESTING
I made similar kind of file as shown above with maximum number of 3 client.Each client provided with different headers.So user has made 3 requests each some specific headers.
To check whether curl loader was able to upload the headers i used wireshark tool for analysing.
for installing wireshark
$sudo apt-get install wireshark
then for running
$ sudo wireshark
then dialog boxes will appear showing error
Just click Ok and use wireshark
now check the request header(Expect: 100-continue, Accept-Encoding: gzip, deflat) for fist request and below screen shot verify that headers were posted for the request.
Thanks for reading.
Please leave your valuable comments.
