Big Data Testing – All you need to know in a nutshell

Big Data is the term that is referred to as Data of Massive in size. This data could be either structured or Unstructured. Today’s most of the organizations owns a wide range of Data and this data plays a crucial role in their success. How Data can be a factor for the “Success” of an Organization? There comes the significance of Big Data and Analytics. Which helps, enterprises to understand patterns, reveal unknowns, describes personas, perspectives of End-users, Products or even the operation of a real-time system.

Big Data has several characteristics. Hence, they defined using 5 V’s.

Data Measurement List to understand how to represent the Size of Data in Big Data World



Why an Organization Requires Big Data testing?

The success of any enterprise depends on how fast they can capture newer markets and Customer base. How can they achieve this? Only the one enterprise that collects information about existing markets, Customers and operations to frame a prototype of new business strategies. These tacts are acquired from various and complex analytics report produced by Big Data. So, ensuring the quality and dependability of Data, that become the core part of a business ‘success’ comes first and foremost challenge for any enterprise. Thus, wins the relevance of Big Data testing in Enterprises.

Big Data process Architecture



1)     Feed data from many sources and analyze the new source of data from sources like social media platforms and Devices sensors.

2)     Process massive volume of Data and make it available to the business for on-demand analytics.

3)     Data Visualization in terms of Dashboards or Metric for further evaluations.

Big Data Testing Scenarios

Accounting the size and components, let us consider scenarios can be performed for Big Data Testing.

Data Ingestion/Staging Stage:

This is considered as the first stage of Big Data process. Here we validate structured or unstructured data taken from various sources mentioned. Data is feed to processing stage using tools Apache NiFi (a.k.a. Hortonworks DataFlow), StreamSets Data Collector (SDC), Gobblin, Sqoop, Flume, Kafka and the list goes on.

Data Processing:

To transform the data from different sources during this stage the one should touch every record incoming. Doing by that ensures the cleanness and standardization of Data for further process. There are couple of tools used at this stage to process. They are frameworks like Hadoop, Spark, Samza, Flink, etc.

Data Storage/Output validation:

This is the validation step, in which Business validates that the output from the big data application is correctly stored in the data warehouse. They also verify that the data is accurately being represented in the business intelligence system or any other target UI.

Performance Testing of Big Data Applications

As Big Data application dealt with a large quantity of Data, it is a must to perform Performance testing to understand Hardware resources utilizations such as Memory, CPU, IO read-write and Data loading and throughput.

People Make the Difference!

Is that true?

YES! ONLY people make the difference.

We have seen different situations handled by different people in different ways. It had shown different results just because the situation was handled by different people. In a world, country, society or family, the situations are managed differently just because it is handled by different people.

Is there any right or wrong?

NO! There is no such right or wrong. Only perceptions judges an act as right and wrong.

How People Make the Difference in an organisation?

People are the backbone of any business. Only if they work together to achieve the goal of an organisation, the Investor get a chance to set next goal. Else there is no scope for next level planning.

Human resource is the supreme resource of an organisation. Rest of the things can buy using money. For that the management need a perfect business structure to fix the backbone (people) in exact place. Only then business can look at the world holding the head straight.

Product and Service of the company is in the hands of employees. If the employee is happy and satisfied, then automatically service excel the expectation.

Here reflects the real meaning of People Make the Difference!
If the people (backbone) is treated well, whole system and body functions perfectly. As a result the organisation goal and customer satisfaction can be achieved easily.
It is true that the collaboration and inclusion of people with diverse backgrounds, experiences and cultures enriches the workplace. It also inspires new insights, ideas and perspectives to full-fill the need.

How does it work in our organisation?

In our work culture we emphasise employee development and a family-like feel with 100% freedom.
We support employees in finding and maintaining excellent work-life balance. Policies are implemented for the development of professionals. Ethics and professionalism is part of our culture.
The resources we hire are matured and experienced enough to behave very practical. They are providing service with dedication and perfection.

Every decision we make is satisfying the foundational values and ethics. These values helps us to have great bonding and relationship with clients.

Yes, we are successful! Our FAMILY Makes the Difference!!!

How to filter LOGCAT log using PROCESS ID filter

Many a times i found it hard to extract log for application that are not debug enabled or cannot be filter with package name ,so the easy way to filter the log is through process id a.k.a PID .The following are few simple steps in getting the process id of the application and applying filter . Note : I have used android studio and make sure “Enable adb integration ” is checked (Tools>>Android>>Enable Adb Integration)

1.Step 1

Open android studio click on android monitor icon

2.Step 2

Connect your device ,make sure adb drivers are installed and also USB debugging is enabled in the device under developer options

3.Step 3

Once this is done your device will get listed in android monitor window ,click on “system information” button in android monitor window and select “Memory Usage” options  . Before that make sure you have opened the application in your device .

4.Step 4

Now in the memory usage dump,find out the application package name (in this case its google maps ) and copy the pid displayed

5.Step 5

Now last and final step is filtering out the log ,Go to the drop down menu displayed top right corner of android monitor window and select Edit filter configuration In the filter window press the + button to add filter and in the PID field provide the previously copied pid and press OK(if you cant to capture the entire log make sure you have selected verbose as log level option )

Now you can see only the application related logs under this filter when you select the same .Happy testing 🙂


© Heidelsoft Technologies Private Limited