owlps-experiments/v1.3.1/README.org

Scenarios


* File naming convention

: sS_jJ_dev_YYYY-MM-DD[_info][+suffix][_similarity].extension

With:
- S :: scenario number.
- J :: test number.
- YYYY :: year.
- MM :: month.
- DD :: day.
- dev :: client device short name.
- info :: an optional informative string can be added, for example when
  a scenario has to be run several times with different parameters.
- suffix :: an optional suffix can be added, separated by a +; suffixes
  have the following meanings:
  + coord :: the real coordinate were added in the file;
  + calib :: the file is a manual calibration file, containing
             calibration requests (type 1) (+coord is implied, as
             manual calibration requests always contain the mobile's
             coordinates); it can be made from real calibration
             requests, or from simple positioning requests with added
             coordinates and adapted type.
- extension :: file extension:
  + agg :: OwlPS Aggregator output file;
  + txt :: experiment report;
  + log :: OwlPS Positioner log file (recorded at the input);
  + pos :: OwlPS Positioner results;
  + out :: OwlPS Positioner standard output;
  + err :: OwlPS Positioner standard error;
  + ods :: results formatted in a spreadsheet.

For the result files, the name of the similarity algorithm used to
compute the positions is added after the suffix. For now it can be one
of the following:
- mean,
- interval,
- interval2.


* Client devices

The following mobile terminals can be used:
- and :: Android smartphone (Samsung Nexus S);
- fon :: Fonera 2.0;
- iph :: iPhone;
- eee :: Asus EeePC 1001-PX (Atheros AR2427 Wi-Fi chipset, Linux
         3.0.0).


* Testing area

** Area description

The deployment area is a room of 5.80 × 10.60 metres. The origin of the
plan is set to the South-West corner of the room.

The East wall is a weight-bearing wall made of concrete, whereas the
others are simple partitions, 9.5 cm thick. The West wall has two doors
and four windows made of Plexiglas. The doors and windows height is 2.5
m.
A West-East room divider built from metal, plastic and wood can be
folded or unfolded to separate the room in two areas of approximately
the same size.

The room is clear from any obstacle, except for the following elements:
- Two technical columns (electricity and network cables) whose diameter
  is 12.5 cm, sitting at the coordinates (1.74;4.72) and (2.46;6.63).
- Another technical column (which is likely to contain water) of floor
  dimensions 31 × 51.5 cm. It sits against the East wall, its centre
  being approximately (2.3;5.65).
- The room divider. Folded its floor dimensions are 115 × 71 cm, and
  its centre is around (5.4;5.7); when it is set up, it splits the room
  at approximately 5.25 m in the Y axis.
- Four heaters (air conditioners) that measure each 150 × 23 cm,
  sitting at each end of the East wall and between the two doors of the
  West wall.
- Two light metal and wood tables and three plastic and metal chairs.

** Measurement points

To simplify the scenario explanation, the following measurement points
are predefined:
1. (5;10)
2. (1;10)
3. (5;1)
4. (1;1)
5. (2.5;5.5)
6. (5;5)
7. (5;2)
8. (3;2)
9. (3;0)
10. (1;5)
11. (1;3)


* Common rules

Except if stated otherwise, all the scenarios follow these rules (or
should follow them for future tests). The description of the scenarios
has precedence over these common rules. Moreover, the report files
associated with each test should also warn about each noticed mistake,
and each exception made to these rules or to the scenario description.

** Devices used

- Mobile terminal: cf. the file names. The OwlPS Client version is
  v1.3.0-11-gc4e0352.
- Aggregation server: Asus EEE-PC 701 4G running Debian GNU/Linux
  squeeze (Linux 2.6.32), with an Atheros AR 500 Wi-Fi interface. It
  runs OwlPS Aggregator v1.3.1-14-ge278aab.
- Listeners: 4 Fonera 2.0, running OwlPS Listener v1.3.0-11-gc4e0352.

** Environmental parameters:

- The temperature is controlled around 22-24°C.
- The humidity varies from 38% to 55%.

Each test's report file should report the temperature and humidity when
the test was started.

** Aggregator

OwlPS Aggregator is run with the default parameters as of the version
used, with autocalibration enabled. These parameters can be found in the
configuration file [[./owlps-config/owlps-aggregator.conf]].

These parameters are not very important, except for the delay between
two autocalibration orders. The default value is 1000 ms.

The default aggregation timeout should fit most test cases, but keep in
mind that for requests as long as one second (including manual
calibration request with the default parameters), it is too short and
must be extended in the Aggregator's configuration.

You should also be aware of the fact that for a capture point to
transmit autocalibration requests, it must be known by the
Aggregator. Therefore, it is not sufficient to wait for 10 seconds
(cf. Measurement procedure) after starting the Aggregator to guarantee
that each capture point transmits autocalibration requests for 10
seconds. The counter must be started after the Aggregator received an
Hello message from all the capture points.

At the end of a test, beware of not stopping the Aggregator too early,
e.g. if requests are sent with long delays, or simply if the aggregate
timeout is not reached for all the requests in memory (as of August
2012, OwlPS Aggregator does not flush the non-aggregated requests upon
exiting, but simply deletes them).

** Listeners

The capture points are attached to the walls and all have their antennas
in vertical position, in the direction of the ceiling.
Their coordinates are given in the OwlPS Positioner's configuration file
([[./owlps-config/listeners-fonera.csv]]).

The OwlPS Listener program runs continuously, with the autocalibration
activated. It is launched with the following command:
: owlps-listenerd -A -v -i 192.168.11.254 -I 192.168.11.254 -r ath1 -w ath0

** Client

The mobile terminal continuously sends positioning requests with the
following parameters:
- 20 packets (-n20),
- 10 ms between two packets (-t10),
- 800 ms between two requests (-F800).
Therefore, one request is transmitted approximately each second.
The destination IP address is the Aggregator's one (i.e. 192.168.11.254
in our setup).

The complete command used to launch OwlPS Client is the following:
: owlps-client -i 192.168.11.254 -n20 -t10 -F800

The mobile terminal's antenna is vertical.

** Measurement-related rules

- Three mobile terminal's altitudes are defined:
  + floor (0 m),
  + hip (0.82 m),
  + ear (1.57 m).
- For the altitudes higher than “floor”, when the mobile terminal is not
  carried by a human operator, it is put on a non-metallic object. In
  our setup, the “hip” altitude is achieved by stacking a empty trash
  (32 cm) on a cardboard box (50 cm); for the “ear” altitude, we add a
  stack of small boxes (75 cm).
- When a human operator carries the mobile terminal, the altitude of the
  terminal is 1 m (hips/belly).
- The antenna of the capture points and the mobile terminal are
  vertical. When the mobile terminal is a laptop computer, the screen is
  vertical, and the computer is set up so that the display side of the
  screen is in direction of the wall opposite to the point where the
  mobile is located (when applicable).

** Measurement procedure

- The infrastructure (Listeners and Aggregator) must be started first
  and at least two rounds of autocalibration request done (i.e. each
  capture point must have sent at least two autocalibration request)
  before the mobile terminal is started. As a best practice, the
  measurements should start at least 10 seconds after the Aggregator
  knows all the Listeners (cf. Aggregator for more details).
- In the scenario in which a human has to move along a path, a metronome
  is set up with the tempo at which the person has to walk. For example,
  60 bpm if the pace is of one metre per second.


* Overview of the scenarios

** Scenario 1

The mobile terminal is still, without human operator, at hip altitude.
Measurements are taken at each corner and the centre of the room
(measurement points 1 to 5), during 1 minute at each position.

** Scenario 2

Repeat the scenario 1, but the mobile terminal is on the floor.

** Scenario 3

This scenario tests the antenna angles and measurement direction, with a
human operator. The measurement points 2 and 5 are tested.

For each point, measurements are taken in two directions, with a 45°
angle (clockwise) between the two directions.
For the measurement point 2, the directions are:
1. East,
2. South-East.
For the measurement point 5, the directions are:
1. North-West.
2. North,

For each direction, three antenna orientations are measured:
1. horizontal,
2. diagonal,
3. vertical.

Therefore, we have 6 measurements per point. Each measurement lasts one
minute.

** Scenario 4

Test with a human operator carrying the mobile terminal. The operator
moves along a path following the measurement points 1 to 5, and stands
at each point for 10 seconds. The pace of the operator is 1 m/s (one
second per step, with one-metre steps).

Timing:
- t-10 :: stand at MP#1 in the direction of MP#2, start the aggregation
          server (with autocalibration activated).
- t0 :: start the client, stay at MP#1 until t10.
- t10 :: start walking to MP#2 (4 m distance).
- t14 :: arrived at MP#2, start rotating in the direction of MP#3.
- t15 :: rotation achieved, stay at MP#2 until t25.
- t25 :: start walking to MP#3 (about 9.85 m distance, so the walk pace
         is around 1.1 m/s to achieve MP#3 in 9 seconds).
- t34 :: arrived at MP#3, start rotating in the direction of MP#4.
- t35 :: rotation achieved, stay at MP#3 until t45.
- t45 :: start walking to MP#4 (4 m distance).
- t49 :: arrived at MP#4, start rotating in the direction of MP#5.
- t50 :: rotation achieved, stay at MP#4 until t60.
- t60 :: start walking to MP#5 (about 4.74 m distance, so the walk pace
         is around 1.2 m/s to achieve MP#5 in 4 seconds).
- t64 :: arrived at MP#5, start rotating to the right (in the direction
         of the mobile wall).
- t65 :: rotation achieved, stay at MP#5 until t75.
- t75 :: stop the client.

** Scenario 5

This scenario aims to evaluate the impact of the delay between the
packets sent by the mobile terminal. The delay between two requests
(-F), must be the same as the delay between two packets (-t).
Each request lasts approximately for one second, and the total number of
packets transmitted during each test must be approximately the same, in
order to ease the statistical treatment.

The following parameters are evaluated:
- 10 ms, 100 packets, 60 requests (6000 packets);
- 20 ms, 50 packets, 120 requests (6000 packets);
- 30 ms, 33 packets, 182 requests (6006 packets);
- 40 ms, 25 packets, 240 requests (6000 packets);
- 50 ms, 20 packets, 300 requests (6000 packets).

The mobile is on the floor, at the centre of the room (2.5;5.5;0).

For each value, the following procedure is run:
1. The aggregation server is launched, and autocalibration requests from
   all the capture points are received for at least 10 seconds.
2. The aggregation server is restarted without autocalibration.
3. The client is run.

Alternatively, if the experiments are done during only one session, the
autocalibration process can be run only once, at the beginning, and the
autocalibration requests stored in a separate file.

Please keep in mind that for requests as long as one second, the
aggregation timeout must be extended in the Aggregator's configuration
(cf. Common rules for details).

** Scenario 6

This scenario uses measurement points 1 and 5-10. For each measurement
point, three altitudes of the terminal are tested: floor, hip and ear.
The room divider is half-closed (from the East wall to the centre of the
room), and there is no human operator in the room.

For each point, each altitude is measured for one minute, in the
following order:
1. floor,
2. hip,
3. ear.

** Scenario 7

Repeat the scenario 6, but with a human operator standing 0.5 m at the
West of the terminal.
Only the hip altitude is studied.

** Scenario 8

Repeat the scenario 7, but the human operator is always standing at the
measurement point 11.

** Scenario 9

This scenario is similar to the scenario 5, but the parameter evaluated
is the size of the packets:
- 64 B,
- 128 B,
- 256 B,
- 512 B,
- 1024 B,
- 1450 B.
This size is the parameter given to OwlPS Client, and is less than the
size of the radio packet. The size of the packet received at the
listeners must be written, as it can vary.

Please refer to the scenario 5 for details and measurement procedure.

** Scenario 10

Repeat the scenario 1 several times, varying the delay between two
packets of the autocalibration requests (option -t of owlps-listenerd):
- 5 ms,
- 10 ms,
- 15 ms,
- 20 ms,
- 25 ms.

** Scenario 11

Repeat the scenario 4 several time with different autocalibration
frequencies on the Aggregator (delay between two autocalibration
orders). The autocalibration requests' settings (number of packets and
delay between two packets) will be adjusted on the Listeners, so that a
request emission lasts for around 20-25 ms less than the autocalibration
frequency.

The following frequencies are tested:
- 100 ms (Listeners: -n10 -t8 = 80 ms),
- 250 ms (Listeners: -n16 -t14 = 224 ms),
- 500 ms (Listeners: -n20 -t24 = 480 ms),
- 1000 ms (Listeners: -n39 -t25 = 975 ms).
*** TODO coord

** Scenario 12

This scenario aims to evaluate the impact of horizontal capture points'
antennas. The scenario 1 is repeated partially (for the measurement
points 3, 4 and 5 only) two times:
1. Each capture point antenna is disposed horizontally, pointing in the
   direction of the opposite wall.
2. The antennas are still horizontal, but placed so that each antenna
   points in the direction of another capture point, in a circle: the
   North-West and South-East listeners' antennas stay in the same
   position as the previous test, but the North-East listener's antenna
   points to the West, and the South-West listener's antenna points to
   the East.

#+CAPTION: Scenario 12, test 1.
#+begin_src ditaa :cmdline -s 2 :file s12-1.png
 +---+ +---+
 |   |        |   |
 |   |        |   |
 +-+-+        +-+-+
   |            |
   |            |
   |            |


   |            |
   |            |
   |            |
 +-+-+        +-+-+
 |   |        |   |
 |   |        |   |
 +---+ +---+
#+end_src

#+CAPTION: Scenario 12, test 2.
#+begin_src ditaa :cmdline -s 2 :file s12-2.png
 +---+ +---+
 |   |        |   |
 |   |        |   |
 +-+-+        +-+-+
   |            |
   |       -----+
   |


                |
   +-----       |
   |            |
 +-+-+        +-+-+
 |   |        |   |
 |   |        |   |
 +---+ +---+
#+end_src

** Scenario 13

This scenario aims to evaluate the impact of the temperature. The
terminal is on the floor, at the measurement point 1. The temperature
starts from a maximum, and lowers to a minimum during the experiment.

** TODO Scenario 14

Repeat the scenario 13, but evaluate the impact of the humidity.

** Scenario 15

Repeat the scenario 1, but without client. The terminal is instead
replaced by a human operator. This scenario aims to evaluate the
influence of the human body on the autocalibration requests.

** Scenario 16

This scenario follows the same principles as the scenario 15, but this
time the scenario 4 is played instead of the scenario 1.

** Scenario 17

Repeat the scenario 16 (scenario 4 without mobile terminal), but with
two human operators, each starting from two opposite corners of the
room (measurement points 1 and 4). They move along the following
measurement points:
- Operator 1: 1, 2, 3, 4, 5 (same as scenario 16).
- Operator 2: 4, 3, 2, 1, 5.

** Scenario 18

The autocalibration is performed for 5 minutes, without mobile terminal
and without human operator.

** TODO Scenario 19

Repeat the scenario 17, but one of the operators carries the mobile
terminal.

** Scenario 20

Manual calibration, without autocalibration running. The mobile device
is carried by a human operator.

Please keep in mind that the default aggregation time is not sufficient
to aggregate correctly manual calibration requests (cf. Common rules).