Performance Testing in Continuous Integration

7 Mar

Continuous Integration

  • Continuous Integration is a practice where a team of developers intermittently merges their code changes into a shared repository. Developers are integrating multiple times throughout the day and ideally committing to changes about once a day or a few times a week.
  • Continuous Integration is preferred by software teams because it allows daily constant feedback in development from all contributors, while it’s easier to catch and resolve bugs early on in the process.




Continuous Integration – Advantages

Detect Performance Degradations When They Happen

As these tests will not be verifying the user experience, we need a different focus. Our strategy is to define a benchmark for a specific version of the system, and then run tests continuously in order to detect a degradation. In a certain way, it’s assuming that the performance that you have today in your infrastructure is okay, and you do not want to miss it when any change negatively affects  this current performance.

Performance Trend Identification

Because we are running the test for every build. We can have track of performance of the application. So that it will help us to maintain the applications performance well under the SLA.



NIOInputStream vs IOInputStream

15 Mar


Main Differences Betwen Java NIO and IO

The table below summarizes the main differences between Java NIO and IO. I will get into more detail about each difference in the sections following the table.

Stream oriented Buffer oriented
Blocking IO Non blocking IO



Java IO’s various streams are blocking. That means, that when a thread invokes a read() or write(), that thread is blocked until there is some data to read, or the data is fully written. The thread can do nothing else in the meantime.

If you have fewer connections with very high bandwidth, sending a lot of data at a time, perhaps a classic IO server implementation might be the best fit. This diagram illustrates a classic IO server design:



Java NIO’s non-blocking mode enables a thread to request reading data from a channel, and only get what is currently available, or nothing at all, if no data is currently available. Rather than remain blocked until data becomes available for reading, the thread can go on with something else.NIO allows you to manage multiple channels (network connections or files) using only a single (or few) threads, but the cost is that parsing the data might be somewhat more complicated than when reading data from a blocking stream.


If you need to manage thousands of open connections simultanously, which each only send a little data, for instance a chat server, implementing the server in NIO is probably an advantage. Similarly, if you need to keep a lot of open connections to other computers, e.g. in a P2P network, using a single thread to manage all of your outbound connections might be an advantage.



What Can Plugins Do in HUDSON

6 Jan

Hudson defines extensibility points, which are interfaces or abstract classes that model an aspect of a build system. Those interfaces define contracts of what need to be implemented, and Hudson allows plugins to contribute those implementations.

One of best features of Hudson is how easy is to extend it. As result there are many plug-ins developed by third parties that can perform a lot of tasks related to the activity of a development team.

Before we proceed with developing the plugin, there is an important concept to understand: How the different project types invoke their plugins.
Currently, one of the biggest source of confusion for end users of Hudson is that most of the plugins do not work with the “m2 project type”. This is because there are essentially two completely different implementations of plugins.

  • Normal plugin
  • Maven plugin

Normal” plugins work with all project types except the  “m2projecttype”, Best performance will be achieved if the heavy lifting is performed on the slave and then finally the results are sent to the master.

Maven plugins only work with the “m2 project type”(m2-maven 2).The plugin author does not have to be as mindful about overloading the master, as by default, they are executing on the slave, however, the side effect is that the plugin does not have full access to all the Hudson objects.

GoAccess: open source real-time web log analyzer/ viewer

6 Jan

GoAccess is an open source real-timeweb log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.

It provides fast and valuable HTTP statistics for system administrators that require a visual server report on the fly

GoAccess was designed to be a fast, terminal-based log analyzer. Its core idea is to quickly analyze and view web server statistics in real time without needing to use your browser (great if you want to do a quick analysis of your access log via SSH, or if you simply love working in the terminal).

While the terminal output is the default output, it has the capability to generate a complete real-time HTML report (great for analytics, monitoring and data visualization), as well as a JSON, and CSV report.

  • Fast, real-time, millisecond/second updates, written in C
  • Only ncurses as a dependency
  • Nearly all web log formats (Apache, Nginx, Amazon S3, Elastic Load Balancing, CloudFront, etc)
  • Simply set the log format and run it against your log
  • Beautiful terminal and bootstrap dashboards (Tailor GoAccess to suit your own color taste/schemes)
  • and of course, Valgrind tested.


netdata – linux monitoring tool (open source)

6 Jan

netdata is a system for distributed real-time performance and health monitoring. It provides unparalleled insights, in real-time, of everything happening on the system it runs (including applications such as web and database servers), using modern interactive web dashboards.

netdata is fast and efficient, designed to permanently run on all systems (physical & virtual servers, containers, IoT devices), without disrupting their core function.



Monitor everything

Analyze thousands of metrics per server. Everything about the system (CPU, RAM, disks, network, firewall, QoS, NFS, etc) and detailed performance metrics for dozens of Linux applications (such as web servers, databases servers, email servers, DNS servers, etc) and SNMP devices.

With alarms

Alarms can be set on any metric monitored by netdata. Alarm notifications are role-based and support dynamic thresholds, hysteresis and can be dispatched via multiple methods (such as email,,,,

In real-time

netdata collects thousands of metrics per server per second, with just 1% CPU utilization of a single core, a few MB of RAM and no disk I/O at all. View everything on stunning real-time interactive web dashboards, even when netdata is running on low-end hardware.

Out of the box

netdata supports auto-detection for everything. It collects more than 5000 metrics automatically, with zero configuration, it has zero dependencies, requires zero maintenance and comes with more than 100 alarms pre-configured to detect common failures, performance and availability issues.


netdata can run anywhere a Linux kernel runs (even IoT) and its charts can be embedded on any web site too.


Custom dashboards can be built using simple HTML (no javascript necessary).


Anything you can get a number for, can be given to netdata, using its Plugin API (anything can be a netdata plugin, BASH, python, perl, node.js, java, Go, ruby, etc).


netdata scales out, your web browser is the central netdata connecting all your serverstogether. netdata can archive its metrics to graphite or opentsdb at a lower rate, to avoid congesting these servers with the amount of data collected.



Why file descriptor metrics are important in performance testing

3 May

A file descriptor is an object that a process uses to read or write to an open file and to open network sockets. An Operating System places limits on the number of file descriptors that a process may open. A lack of available file descriptors can cause a wide variety of symptoms which are not always easily traced back to. The Open Files Descriptors (OFD) provides a count of the total number of file descriptors that are currently allocated and open for processing. The percentage of the total number of open file descriptors with respect to the maximum allowed count of descriptors for processing is a good metric for evaluating the health of a web application
A file descriptor is an opaque handle that is used in the interface between user and kernel space to identify file/socket resources. Therefore, when you use open() or socket() (system calls to interface to the kernel), you are given a file descriptor, which is an integer (it is actually an index into the processes u structure – but that is not important). Therefore, if you want to interface directly with the kernel, using system calls to read(), write(), close() etc. the handle you use is a file descriptor.

There is a layer of abstraction overlaid on the system calls, which is the stdio interface. This provides more functionality/features than the basic system calls do. For this interface, the opaque handle you get is a FILE*, which is returned by the fopen() call. There are many many functions that use the stdio interface fprintf(), fscanf(), fclose(), which are there to make your life easier. In C, stdin, stdout, and stderr are FILE*, which in UNIX respectively map to file descriptors 0, 1 and 2.



Step # 1 Find Out PID

To find out PID for mysqld process, enter:
# ps aux | grep mysqld
# pidof mysqld


Step # 2 List File Opened By a PID # 28290

Use the lsof command or /proc/$PID/ file system to display open fds (file descriptors), run:
# lsof -p 28290
# lsof -a -p 28290

# cd /proc/28290/fd
# ls -l | less

You can count open file, enter:
# ls -l | wc -l

Tip: Count All Open File Handles

To count the number of open file handles of any sort, type the following command:
# lsof | wc -l
Sample outputs:


Emptying the buffers cache in linux

11 Mar

If you ever want to empty it you can use this chain of commands.

# free && sync && echo 3 > /proc/sys/vm/drop_caches && free

total used free shared buffers cached
Mem: 1018916 980832 38084 0 46924 355764
-/+ buffers/cache: 578144 440772
Swap: 2064376 128 2064248
total used free shared buffers cached
Mem: 1018916 685008 333908 0 224 108252
-/+ buffers/cache: 576532 442384
Swap: 2064376 128 2064248
You can signal the Linux Kernel to drop various aspects of cached items by changing the numeric argument to the above command.

To free pagecache:

# echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:

# echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:

# echo 3 > /proc/sys/vm/drop_caches
The above are meant to be run as root. If you’re trying to do them using sudo then you’ll need to change the syntax slightly to something like these:

$ sudo sh -c ‘echo 1 >/proc/sys/vm/drop_caches’
$ sudo sh -c ‘echo 2 >/proc/sys/vm/drop_caches’
$ sudo sh -c ‘echo 3 >/proc/sys/vm/drop_caches’