LINK BETWEEN RESEARCH QUESTIONS AND METHOD

Get Complete Project Material File(s) Now! »

Research Method

The goal of this chapter is to describe and motivate the general work process. In this chapter the research method will be described and motivated.

Link between research questions and method

To answer the research questions of this thesis, it is important to consider how the questions should be answered and what empirical data would be needed. This is important when considering what research design and data collection should be used to answer the questions.
The first research question, ”How does MQTT impact latency and power consumption on a constrained device?”, requires quantitative empirical (primary or secondary) data that describes the latency and power consumption impact MQTT have on a constrained device. However, since MQTT have QoS settings that influences the amount of acknowledgement messages sent, it would be interesting to see how this behavior influences the latency and power consumption of MQTT.
The second research question, ”How does CoAP impact latency and power consumption on a constrained device? ”, similar to the first research question, requires quantitative empirical data that describes the latency and power consumption impact CoAP have on a constrained device. However, since CoAP is a UDP based protocol and lack any QoS levels, it is unsuitable to compare to MQTT as they are different from each other. Therefore, it is important that they are not compared to each other in the analysis, as it would be difficult to know if it is the TCP part of MQTT that impact the latency or power consumption.
The third and last research question, ”How does usage environment influence the latency for MQTT and CoAP?”, requires quantitative empirical data that describes the latency impact MQTT and CoAP have on a constrained device in different environments where the Wi-Fi signal strength is varying. This data can be acquired, while answering the first and second research question. However, the data needs to be collected from at least two different environments.
To find answers to our research questions, it is necessary to obtain the aforementioned data. However, there are several ways to obtain the data. Wohlin et al. book ”Experimentation in software engineering” [25] give insight in what research method as well as data techniques are suitable for the task. For example, by performing a literature review where already existing data from other theses is used or reviewed to answer the research questions. But, this approach requires the data to exist in the first place, and if that data exist, it must be related to the data which it is being compared to (with respect to the goal of the thesis) in some ways [25]. However, if the desired data does not exist, the researcher may need to devise a plan for obtaining such data.
In general, this can be done through experiments, surveys and case studies. Of course, there are no real silver bullets when it comes to selecting a data collection technique, since some techniques might be more favorable than others, depending on the goals or entities the studies intend to involve.
Experiments can be suitable for both human-oriented as well as technology-oriented research, where the experiment can be a true, quasi, natural, field or controlled experiment, depending on what is suitable for the research goal [25].
Surveys, just like experiments, also come in several types or designs which have their own purpose. The key difference between experiments and surveys is that survey methods are more suitable for obtaining data that requires human interaction (people-oriented research) [25].
Case studies on the other hand, can utilize all data collection methods deemed necessary, which can strengthen the scientific reasoning by utilizing triangulation (obtaining data from multiple methods). Case studies, just like the other methods mentioned, come in many types and designs which, again, have their own purposes. Such as exploratory, illustrative, cumulative and critical instance case studies [25]. However, case studies are mostly suitable for research questions that handles a specific case that requires deep knowledge to be answered or handles problems that are hard to understand or define.
Out of the methods explained here, case study seems like a compatible choice for this thesis. However, not so much for the goal of this thesis since the goal is more focused on a cause and effect relationship rather than being focused on a specific case or an event. Furthermore, since this thesis is more concerned with providing quantitative data about the latency and power consumption impact communication protocols (such as MQTT and CoAP) have on a constrained device, it seems more logical to analyze the interaction between the code and the machine involved. Therefore, out of the 3 methods, experiments seem like the most suitable method as it fits the goal of the thesis better and it provides more control over the data collection process [25], which may be helpful when answering the third research question.

Experimental research design

This thesis has used and extended the experiment process described in the ”Experimentation in software engineering” book by Wohlin et al. [25]. The reason why Wohlin et al. experiment process was selected is because it was deemed the most suitable and robust for the data collection method for the purpose of this study.
Wohlin et al. [25] experiment process consists of 5 steps that guide the researcher through the research process, where the process in question requires an experiment idea. Wohlin et al. process ends with a report that describes the key points of the research. Wohlin et al. experiment process can be seen in Figure 5.
While Wohlin et al. experiment process seems robust, it still requires an experiment idea as a pre- requisite for the overall process. Therefore, it was decided to extend Wohlin et al. process by adding 5 steps in the beginning of the process to make it reflect how the authors conducted their work and how this pre -requisite was attained. The overall extended process can be seen in Figure 6, where the blue tinted boxes are part of Wohlin et al. [25] process and the yellow boxes are the activities added to make this process more suitable for this thesis.
As seen in Figure 6, it consists of 11 activities, where the first 5 steps involve the discovery of the research problem and questions as well as deciding on what research methods should be used to answer these questions. The remaining 6 steps (step 6 – 11) is from Wohlin et al. [25] process (as seen in Figure 5) and is related to the experiments themselves and how it should be conducted.
These activities will be explained in their appropriate sections where sub-section 3.2.6 to 3.2.10 describe an abridged version of Wohlin et al. research activities.
Wohlin et al. [25] state that it is important to understand that these activities, as well as their sub-activities, does not necessarily need to be executed in a linear fashion (or to that of a waterfall mentality). Especially since new knowledge or problems could be unveiled as the researcher work with the process, which can make some earlier designs, goals and other actions obsolete, trivial or impossible to conduct.

Find topic

This activity is focused on finding research topics as well as exploring research domains.
The reasoning behind adding this activity as a part of the process is because it is a common activity that the researcher is involved in when trying to find a suitable research domain, but it also helps the researcher to see intersecting research domains which can be helpful when doing the background research.

Background research

The purpose of this activity is to help the researchers understand the research domain better but also to help them to find a suitable research problem.
This activity usually involves learning about the domain (be it process, technology or human focused research) and to find the state of the art in this domain which also allows the researcher to better understand the limits of the domain and where the research can be done. This also help the researcher selecting a research method.

Spike research goal

The prerequisites for this activity is that a research objective, goal or problem has been identified, which will then be spiked (declared as the main reason for this research) in this activity. When the research goal has been spiked it should be considered as the main goal or purpose of the research. The overall goal of this activity helps the researchers to keep the thesis focused as well as defining the scope.

Formulate research questions

When the research goal or problem is defined, one or several research questions (as well as sub research questions) can then be defined to further focus the study, but also to assist the researcher to segment the research problem if it is hard to understand or rather complex.

Explore research designs and data collection methods

When the research questions, goals and problems have been outlined and understood, the data collection method and research design can be decided upon. When deciding on a research design or a data collection method, it is necessary to consider what kind of empirical data is needed to answer the research questions as well as what research designs may be suitable for the goal of the research. In this activity the researchers reflect upon what data is necessary to answer the research questions and how this data can be acquired.

Experiment scoping

According to Wohlin research process, this research activity explores the scope of the experiments (not to be confused with the scope of the overall research). This involves: deciding upon what is supposed to be the unit of analysis, the purpose of the experiment, from what perspective these experiments are performed, what aspect the experiments are focusing on, what perspective the experiments are trying to relate to and the context of the experiment and the research.
To achieve this, Wohlin et al. [25] suggests a template that will help researchers scoping their experiments. This template looks like this:
”Analyze <Object(s) of study>
For the purpose of <Purpose>
with respect to their <Quality focus>
from the perspective of the <Perspective>
in the context of <Context>.
” – Wohlin et al. [25], page 85
For the sake of clarity each scoping entity from the template is listed and described below based on how it is described in Wohlin et al. [25].
Object(s) of study
This determines what should be studied, be it a product, process, model, metric or theory. Determining what should be studied will help the researchers to design their experiments within the scope of the research.
Purpose
The purpose details what the experiments are trying to achieve on the object of study. Such as to characterize, monitor, evaluate, predict, control or change something about the object of study.
Quality Focus
The quality focus is related to the purpose in the sense that the researchers usually wants to have a focus to the purpose. It can be considered as the quality requirements. Examples of quality focus could be effectiveness, cost, reliability, maintainability, portability, etc.
Perspective
The perspective explains the viewpoint or actors (i.e. professional or researcher) this research has. This is defined to give a better understanding about the quality focus in question, as its details can greatly depend on the actors involved. For example, some quality focuses have different meanings when it comes to different perspectives, like reliability for a user versus a developer.
Context
Defining the context gives an understanding on how the research is performed and by whom. The context helps the readers to understand the limits of the project but also where the researchers come from which can help the readers to understand the scope better, but also how the research will be conducted.
When all these entities are defined, they will act as the base foundation for the planning activity.

Experiment planning

According to Wohlin research process, the goal of this activity is to create a plan for how the research goal can be achieved, where the plan usually follows the scope of the research. Thus, it is necessary to decide upon what kind of experiments will be conducted, what hypotheses will be tested, which variables will be used for the experiments, which subjects will be used, how the experiments will involve the variables and subjects, how the desired or effect data will be collected or measured and lastly if there are any validity concerns with the proposed plan and how these concerns can be acted upon.
Wohlin et al. [25] divides the planning activity into 7 sub activities where each sub-activity handles certain aspects of the experiment planning. These sub-activities are:
Context selection
The goal of this sub-activity is to decide on what type of experiment will be performed. For example, is it done off-line or on-line, by students or professionals, does it handle toy problems (unrealistic or uncommon problems) or real problems and if the experiments are done for a general or specific problem. The context can also describe how general the experiments seeks to be.
Hypothesis formulation
The goal of this sub-activity is to formulate a hypothesis or several hypotheses that can be supported or disproven by the collected data.
Wohlin et al. [25] suggest that a hypothesis can be thought as the formal definition for an experiment. Formulating hypotheses is important as they will be the foundation for the experiment design, since they describe the conditions and actors or objects in relation to the research goal or problem.
Variables selection
In this sub-activity the dependent and independent variables are selected taking both the hypothesis as well as the goal of the research. Selecting good variables is important since it can make the results clearer and strengthen the analysis, where bad variables can lead to unclear or misleading results.
Subject selection
The goal of this sub-activity is to determine what samples (or subjects) are most suited for the overall goal of the research. The researcher decides on the sampling size, sample population and the sampling techniques to be used when performing the experiments.
Experiment design
The goal of this sub-activity is produce a preliminary experiment design which is based on the conclusions made in the previous sub-activities. This design is thought of as the results of these conclusions and should be designed to either support or disprove the hypotheses with respect to the variables used, the subjects included and in the given context with as high validity as seen fit. A good experiment design can also strengthen the analysis by removing possible anomalies or unwanted behavior that could impact the experiments.
Instrumentation
The goal of this sub-activity is to decide on what objects will be used in the experiment, what procedure (guidelines) these objects will follow and how the data collection will be performed as well describing what is required for the data collection to work with respect to the selected variables.
Validity evaluation
The goal of this sub -activity is to identify and prevent threats to the validity that might come with the experiment design.

Experiment operation

In Wohlin research process, the experiment operation is carried out when the planning for the experiments has reached a certain maturity level. This activity is dedicated to the overall operation of the experiments, where this activity has 3 sub-activities that handles the experiment preparation, execution and data validation.
Preparation
The preparation sub-activity is where the researcher setup the experimental environment, gather the subjects or objects used in the experiment and by preparing the instruments. When the experiment setup is sufficiently prepared the researcher can proceed to execute the experiment.
Execution
The execution sub-activity can simply be described as performing the experiments as well as the data collection according to the plan. When the data has been collected the researcher can proceed to perform the data validation.
Data validation
The goal of this sub-activity is to make sure that the collected data is valid, since the data that has been collected can contain a lot of anomalies that might not seem as reasonable output for the experiment. This can be caused by a lot of things. By checking the data, the researcher can detect potential problems or inconsistencies with the experiment setup and perform corrective actions to fix these errors.

Experiment analysis and interpretation

In Wohlin research process, when the experiment data has been acquired it can then be analyzed to find casual relationships in the collected data as well as giving a statement on the given hypothesis. Depending on the type of data (be it quantitative or qualitative) some analysis techniques are more suited than others as well as ways of present the analyzed data. This activity has 3 sub-activities which handle the descriptive aspect of the gathered data, data set reduction and lastly hypothesis testing.
Descriptive statistics
The goal of this sub-activity is to consider what data should be displayed and how the data should be visualized, so that any conclusions and significant results are visible for both the researcher and the readers of this paper.
Data set reduction
The goal of this sub-activity is to handle potential outliers or anomalies found within the data set. An outlier can either be removed or kept for the analysis depending on the goal of the research. Some outliers can impact how the results are perceived if, for example, the average is used to explain an element in the data set, since an overall high data value can devalue the effect of the average and thus make the conclusion misleading or incorrect.
Hypothesis testing
The goal of this sub-activity can simply be described as deciding if the hypothesis is supported or disproven and how valid this conclusion is depending on the goal as well as the context of the research considering, also the design of the experiments.

Presentation and packaging

According to Wohlin et al. the goal of this activity is to structure the report which contains the findings and conclusions as well as the implementation of the research so that the content within is appropriate for its intended readers.

Implementation

This chapter describe how the research method is applied, hypotheses and experiment design used for the experiments.

Work flow

This section is dedicated to describing how the research method was applied. This is done by mapping the overall activities to the chapters of this study.
The first activity, find topic was carried out by exploring trending topics within the software engineering and information technology discipline. The topic was selected by observing technology focused news websites and from the authors (of this study) own experience. Where IoT was selected as the overall topic of the study. The topic went through multiple iterations from being focused on IoT to performance aspects of constrained device communication. This iteration is a result of the background research activity. The results from this activity are the basis for the content within chapter 1 as well as the theme for the whole study.
The second activity, background research was carried out by finding the state of the art (or current problems) in relation to the topic. This was done by searching for literature related to the topic. Where Google scholar, DiVA portal, ACM digital library and IEEE Xplore digital library were used as the search engines for the literature. Additionally, the snowballing method was used to find other literature. The results from this activity are the basis for section 1.1 and 2.2.
The third and fourth activity, spike research goal and formulate research question was done as a result from the background research activity, where the problem domain has been explored and the current problems have been raised by other researchers. This lead to the research goal being focused on power consumption and latency aspects related to network communication of a constrained device. Where research questions were formed to handle these performance aspects. The results from these activities are shown in section 1.2.
The fifth activity, explore research designs and data collection methods, was carried out by examining the research questions and research problem on how to best answer it. The research designs were selected by reading about research methods and their suitability in research method focused books, like Wohlin et al [25]. It was decided to extend Wohlin’s research process. The result from this activity is shown in chapter 3, where section 3.1 describe and motivate the choice of research designs. Section 3.2 describes Wohlin et al. research process as well as the extended research process.
The sixth activity, experiment scoping, was carried out by mapping the purpose and research goal to the template provided by Wohlin et al. This limited the scope of the study and the experiments, which also focused experiment purpose to be focused on latency connectivity performance aspect. The results from this activity is shown in section 1.2 and 1.3, but also in sub-section 4.3.1 where the template is used to describe the experiment scope.
The seventh activity, experiment planning, was carried out by creating hypotheses and conducting prototype experiments. The hypotheses were created with respect to the experiment scope, purpose of the study and previous research. The prototype experiments tested the limitations of the hardware, sites, radio frequency transmitters, communication protocols and code frameworks so it was clear if the experiments should be conducted on that that setup. This activity resulted the content seen within section 4.2 and 4.3. Section 4.3 list and describe how the experiments were conducted.
The eight activity, experiment operation, was carried out by readying the environments for the experiments where Wi-Fi signal strength was collected from the environments (this was done in parallel with the previous activity) and was carried out by executing the experiment plan that was created in the previous activity. After this was done the clients, brokers and servers were setup and then the experiments are carried out. The Wi-Fi signal is recorded in section 4.4 for each of the environments and the results from the experiments can be found in the findings section 5.1.
The ninth activity, experiment analysis and interpretation , was carried out by examine the data collected from the previous activity where the validity evaluation as described in sub- section 4.3.6 was used to improve the result. The results from the analysis is shown in the section 5.1.2 and the conclusion in chapter 6.
The last activity, presentation and packaging, was carried out throughout the study.

Hypothesis formulation

This section is dedicated to describing the hypotheses as well as motivating why these hypotheses has been formulated.

Research question 1

For the first research question, ”How does MQTT impact latency and power consumption on a constrained device?”, two hypotheses are defined to handle the latency and power consumption impact aspect for MQTT.

Hypothesis 1

To understand how MQTT and its QoS levels influences the latency, the following hypothesis is defined:
H1: « Using Higher QoS levels in MQTT should result in a higher latency since there are more messages sent before the message transaction is considered complete »
This hypothesis is based on the results seen in Lindén’s [7] research, where the results show that the latency is higher for higher QoS levels. The hypothesis is defined to test if the observation seen in Lindén’s [7] research applies to lower message sizes as well, and in a sensor node to gateway scenario.
For the hypothesis to be supported, the experiments should clearly show that the QoS levels increase the latency, regardless of what message sizes are used. However, for the hypothesis to be disproven the experiment results need to show that the QoS levels do not increase or have an impact on the latency at all.
Regardless if the hypothesis is supported or disproven, the result will still be able to answer the first research question. For instance, if the hypothesis is supported it would imply that the QoS levels for MQTT increases the latency. If the hypothesis is not supported (disproven) it would imply that higher QoS levels does not increase the latency for smaller message sizes, which also answers the latency aspect of the first research question.

Contents
1 Introduction
1.1 BACKGROUND
1.2 PURPOSE AND RESEARCH QUESTIONS
1.3 DELIMITATIONS
1.4 OUTLINE
2 Contextual Framework
2.1 TECHNICAL FRAMEWORK
2.2 THEORETICAL FRAMEWORK
3 Research Method
3.1 LINK BETWEEN RESEARCH QUESTIONS AND METHOD
3.2 EXPERIMENTAL RESEARCH DESIGN
4 Implementation
4.1 WORK FLOW
4.2 HYPOTHESIS FORMULATION
4.3 EXPERIMENT DESIGN OVERVIEW
4.4 EXPERIMENT PREPARATION
5 Findings and analysis
5.1 FINDINGS
5.2 ANALYSIS
6 Discussion and conclusions
6.1 DISCUSSION OF METHOD
6.2 DISCUSSION OF FINDINGS
6.3 CONCLUSIONS
6.4 FUTURE WORK
7 References
GET THE COMPLETE PROJECT