owlps/scripts/owlps-aggsetcoord.pl

#!/usr/bin/perl -w

=head1 NAME

owlps-aggsetcoord - set the coordinates in an aggregation CSV file


=head1 SYNOPSIS

B<owlps-aggsetcoord>
[ B<-h> | B<-V> ]
[ B<-v> ]
[ B<-i> | B<-I> I<suffix> ]
[ B<-c> I<coordinates> [ B<-l> I<line_selection> ] ]
[ B<-m> I<mac> ]
[ I<aggregation_file> ]


=head1 DESCRIPTION

B<owlps-aggsetcoord> is a tool to set the coordinates of mobile devices in a CSV
aggregation file, in an automated way. It works only with lines containing
positioning requests; the lines containing other types of requests are left
untouched. Furthermore, only the default values of the coordinates will be
treated; a non-default coordinate will always be left untouched. Note that the
default coordinate in OwlPS is registered in aggregation files as "0.00",
therefore this program considers the string "0.00" as the default value
corresponding to an undefined coordinate. Zero coordinates are not prevented,
but you have to write them in another way in your input file, such as "0",
"00.00", "0.0", "0.000", etc.

By default, B<owlps-aggsetcoord> uses the coordinates already present in the
input file (manually set by the user, most of the time) to interpolate the
coordinates of lines containing default values. A given coordinate is
interpolated according to the previous and next known coordinates and their
timestamps, assuming the mobile moves at a constant speed between two known
coordinates. The interpolation is done in 3 dimensions, each dimension being
interpolated independently.

Alternatively, using the B<-c> option will disable interpolation but apply an
arbitrary set of coordinates to the lines containing only default values.

In theory, you can provide more than one I<aggregation_file> at once, but don't
do that unless you really know what you're doing; you cannot specify multiple
input files when using the B<-i> option. The (list of) aggregation file name(s)
I<must> be placed after the options. If no file is provided, the standard input
is read.

Unless B<-i> is used, the results are written on the standard output. It is
advised to use graphical diff tools such as I<vimdiff> to visualise the
differences between the input file and the result and make sure it corresponds
to what was expected.


=head1 OPTIONS

=over 7

=item B<-h>, B<--help>

Print version and help message and exit.

=item B<-V>, B<--version>

Print version message and exit.

=item B<-v>

Turn on verbose mode (displays a trace on the standard error).

=item B<-i>

Modify the input file in-place. The original file is saved with suffix ".orig".

=item B<-I> I<suffix>

Same as B<-i>, but the original file is suffixed with I<suffix> instead of
".orig". If both B<-i> and B<-I> are used, the latter has precedence over the
former. If I<suffix> is an empty string, this option is ignored.

=item B<-c> I<coordinates>

Apply I<coordinates> to every positioning request with default X, Y and Z. A
line in which at least one of the three coordinates is set to a non-default
value will be left untouched. I<coordinates> is a string "X;Y;Z".

=item B<-l> I<line_selection>

Skip lines that are not part of the selection pattern I<line_selection>, which
is evaluated as a Perl numeric list. For example, "3, 5, 8..10" will select the
lines 3, 5, 8, 9 and 10; "1..25,42..45" will select the lines 1 to 25 and 42 to
77. This works only with the B<-c> option.

=item B<-m> I<mac>

Work only on positioning requests transmitted by MAC address I<mac>.

=back


=head1 BATCH PROCESSING

If you have a bunch of different coordinates to put in the same aggregation file
(named F<input.agg> in the following example), you can prepare a file that
contains, on each line, a line range and the set of coordinates to apply at this
line range, separating the two fields by a space. You can then use a little
shell script such as the following to read your input file (named
F<batchfile.txt> here) and call AggSetCoord for each line:

    FILE=input.agg
    BATCHFILE=batchfile.txt
    i=0
    while read LINES COORD ; do
        i=$(expr $i + 1)
        echo "Setting coordinates ($COORD) at line(s) $LINES"
        owlps-aggsetcoord -I .step$i -l "$LINES" -c "$COORD" "$FILE"
    done <"$BATCHFILE"

You can also include the MAC address in the input file and call AggSetCoord with
the B<-m> option.


=head1 COPYING

This script and its documentation are part of the Owl Positioning System (OwlPS)
project. They are subject to the copyright notice and license terms in the
COPYRIGHT.t2t file found in the top-level directory of the OwlPS distribution
and at https://code.lm7.fr/mcy/owlps/src/master/COPYRIGHT.t2t


=head1 SEE ALSO

owlps(7), owlps-aggregatord(1)

=cut


use strict;
use Getopt::Std;
use Pod::Usage;
use File::Temp qw/tempfile/;
use File::Basename qw/dirname/;
use File::Copy qw/move/;
use OwlPS::TimeInterpolation;
use OwlPS::CSV;


## Constants ##

# CSV format version handled by this program
use constant FORMAT_VERSION => 1;

# Indexes of the fields in the CSV lines (first field is #0)
use constant MAC_FIELD_INDEX       => 1;
use constant TYPE_FIELD_INDEX      => 2;
use constant TIMESTAMP_FIELD_INDEX => 4;
use constant X_FIELD_INDEX         => 5;
use constant Y_FIELD_INDEX         => 6;
use constant Z_FIELD_INDEX         => 7;

# Number corresponding to a normal positioning request
use constant REQUEST_TYPE_NORMAL => 0;

# String corresponding to an "empty" coordinate in the input file
use constant BASE_COORD => "0.00";


## Global variables ##

# Output file handle
my $output_handle = *STDOUT;

# Names of the temporary and final output files
my ($tmp_output_file, $output_file);

# Suffix with which the original file is saved
my $backup_suffix;

# Verbose mode
my $verbose;

# List of the lines to work on (will stay undefined if all the lines have to be
# worked on)
my %selected_lines;

# Selected transmitter's MAC address (will stay undefined if -m was not used)
my $selected_mac;

# New coordinates to apply to the selected lines (will stay undefined if -c was
# not used)
my $new_coordinates;
my ($new_x, $new_y, $new_z);

# Current line
my $cur_line;

# Current line number
my $line_nb = 0;

# Last "real" values of X, Y and Z to interpolate with
my ($last_x, $last_y, $last_z);

# Timestamps corresponding to each of these last values
my ($last_x_t, $last_y_t, $last_z_t);

# Number of waiting Xs, Ys and Zs that we will have to interpolate
my ($xs, $ys, $zs) = (0, 0, 0);

# Array of the previous lines to interpolate (we store them while we are waiting
# for the next "complete" line)
my @prev_lines;


## Functions ##

sub VERSION_MESSAGE {
    my $handle = $_[0] || *STDOUT;
    print $handle
      "This is OwlPS AggSetCoord, part of the Owl Positioning System project.\n"
      . "CSV aggregation format version handled: "
      . FORMAT_VERSION . "\n";
}


sub HELP_MESSAGE {
    my $handle = $_[0] || *STDOUT;
    pod2usage(-output  => $handle,
              -exitval => "NOEXIT");
}


# Prints a trace message if verbose mode is active (no new line is printed).
# Parameters: @message
sub trace(@) {
    print STDERR "Line #$line_nb: @_" if ($verbose);
}


# Prints a warning message terminated by a new line.
# Parameters: @message
sub print_warning(@) {
    print STDERR "Warning on line #$line_nb: @_\n";
}


# Prints the arguments to the output handle.
sub output(@) {
    print $output_handle @_;
}


# Returns true if $mac is the transmitter's MAC address selected by the user, or
# if the user didn't select any particular MAC address.
# Parameters: $mac
sub is_selected_mac($) {
    if (!defined($selected_mac)) { return 1 }
    return $selected_mac eq $_[0];
}


# Returns true if all $x, $y and $z are ready to be printed.
# Arguments: $x, $y, $z
sub is_complete($$$) {
    my ($x, $y, $z) = @_;
    return ($x ne BASE_COORD && $y ne BASE_COORD && $z ne BASE_COORD);
}


# Returns true if all $x, $y and $z are untouched (fully incomplete line) or
# undefined.
# Arguments: $x, $y, $z
sub is_blank($$$) {
    my ($x, $y, $z) = @_;
    return (!defined($x) && !defined($y) && !defined($z))
      || ($x eq BASE_COORD && $y eq BASE_COORD && $z eq BASE_COORD);
}


# Checks the stored previous line and prints/delete what the complete lines,
# starting with the oldest line stored and stopping with the first incomplete
# line.
sub check_oldest_lines() {
    while ($prev_lines[0]) {
        # If the oldest stored line is empty, just print it and delete it
        if ($prev_lines[0] eq "\n") {
            output(shift @prev_lines);
            next;
        }

        # Check the oldest (non-empty) line
        my (
            $csv_format,
            $mac,
            $type,
            $nb_pkt,
            $timestamp,
            $x,
            $y,
            $z,
            $end_line
        ) = split(';', $prev_lines[0], 9);

        # Display and delete the line if either of the following is true:
        # - it is not a positioning request
        # - it is not the selected transmitter's MAC address
        # - it is complete
        if (   $type != REQUEST_TYPE_NORMAL
            || !is_selected_mac($mac)
            || is_complete($x, $y, $z))
        {
            output(shift @prev_lines);
            next;    # Jump to the next oldest line
        }

        # If this line is not complete, it will have to be interpolated and the
        # potential newer stored lines as well, so we end the check here
        last;
    }
}


# Modifies the field number $idx of $nb_lines previous lines stored in
# @prev_lines with values interpolated from real coordinates $start and $stop
# registered at $start_time and $stop_time. Lines containing non-positioning
# requests are skipped and do not count in $nb_lines.  Parameters: $nb_lines,
# $idx, $start, $start_time, $stop, $stop_time
sub apply_interpolation($$$$$$) {
    my ($nb_lines, $idx, $start, $start_time, $stop, $stop_time) = @_;

    for (my $i = 1 ; $i <= $nb_lines ; $i++) {
        # Skip blank lines
        if ($prev_lines[-$i] eq "\n") {
            # Incrementing the number of lines to treat has no incidence but to
            # allow the loop to go one element further to take into account the
            # blank line
            $nb_lines++;
            next;
        }

        # Skip non-positioning requests
        my $type = get_field($prev_lines[-$i], TYPE_FIELD_INDEX);
        if ($type != REQUEST_TYPE_NORMAL) {
            $nb_lines++;
            next;
        }

        # Skip requests sent by other devices than the selected one
        if (defined($selected_mac)) {
            my $mac = get_field($prev_lines[-$i], MAC_FIELD_INDEX);
            if ($selected_mac ne $mac) {
                $nb_lines++;
                next;
            }
        }

        my $time = get_field($prev_lines[-$i], TIMESTAMP_FIELD_INDEX);
        my $newvalue =
          interpolate_1d($start, $start_time, $stop, $stop_time, $time);
        # Round the interpolated coordinate to two decimals
        $newvalue = round2($newvalue);
        # But if the new value is "0.00", it will be considered as "incomplete",
        # so we add a leading 0 to avoid being confused when we later check the
        # line
        $newvalue = "0" . $newvalue if ($newvalue == BASE_COORD);
        $prev_lines[-$i] = update_field($prev_lines[-$i], $idx, $newvalue);
    }
}


## Option parsing ##

$Getopt::Std::STANDARD_HELP_VERSION = 1;
use constant OPTIONS => 'c:hiI:l:m:vV';
my %options;
if (!getopts(OPTIONS, \%options)) {
    HELP_MESSAGE(*STDERR);
    exit 1;
}

if ($options{'h'}) {
    VERSION_MESSAGE();
    HELP_MESSAGE();
    exit 0;
}

if ($options{'V'}) {
    VERSION_MESSAGE();
    exit 0;
}

$verbose         = $options{'v'};
$new_coordinates = $options{'c'};
$backup_suffix   = $options{'I'};
$selected_mac    = $options{'m'};

if ($options{'i'} || $backup_suffix) {
    if (@ARGV != 1) {
        die "One and only one file name must be provided when using -i or -I";
    }
    $backup_suffix ||= ".orig";
    $output_file = $ARGV[0];
    ($output_handle, $tmp_output_file) =
      tempfile(
               ".owlps-aggsetcoord-XXXXXXX",
               SUFFIX => ".agg",
               DIR    => dirname($output_file),
               UNLINK => 1
      );
}

if (defined($new_coordinates)) {
    ($new_x, $new_y, $new_z) = split_point_3d($new_coordinates);
}

# Register the selected lines
if (defined($options{'l'})) {
    die "-l option requires -c option!" if (!defined($new_coordinates));
    $selected_lines{$_} = 1 foreach eval $options{'l'};
}


## Main loop: read lines ##

while ($cur_line = <>) {
    $line_nb++;
    trace($cur_line);


    # The current line is empty
    if ($cur_line eq "\n") {
        # If we don't have previous lines, we print it; if we do, we store it
        if   (@prev_lines == 0) { output("\n") }
        else                    { push @prev_lines, "\n" }
        next;
    }


    my (
        $csv_format,
        $mac,
        $type,
        $nb_pkt,
        $timestamp,
        $x,
        $y,
        $z,
        $end_line
    ) = split(';', $cur_line, 9);

    # Check the CSV format version
    if ($csv_format != FORMAT_VERSION) {
        die "CSV format \"$csv_format\" not supported!";
    }


    # Check the stored previous line and print what we can
    check_oldest_lines();


    # Check the request type: we work only on positioning requests but must take
    # into account other request types
    if ($type != REQUEST_TYPE_NORMAL) {
        trace("Non-positioning request: line skipped.\n");
        # If we have no lines in memory, we can just print the line and jump to
        # the next one, otherwise we have to store the line
        if   (@prev_lines == 0) { output($cur_line) }
        else                    { push @prev_lines, $cur_line }
        next;
    }


    # Check the transmitter's MAC address
    if (!is_selected_mac($mac)) {
        trace(  "Transmitter's MAC address ($mac) doesn't match the selected"
              . " transmiter's MAC address ($selected_mac): line skipped.\n");
        # Print or store the line
        if   (@prev_lines == 0) { output($cur_line) }
        else                    { push @prev_lines, $cur_line }
        next;
    }


    # The user specified a coordinate to apply to every pertinent line (-c
    # option): we print it immediately (we are sure not to have any previous
    # lines in memory, since we don't do that in this mode)
    if (defined($new_coordinates)) {
        # Skip the line if it is not part of the selected lines
        if (%selected_lines and !$selected_lines{$line_nb}) {
            trace("Line not selected: skipped.\n");
            output($cur_line);
            next;
        }

        # If the current line is fully incomplete, we reconstitute it with the
        # new coordinates
        if (is_blank($x, $y, $z)) {
            output(
                   $csv_format, ";",
                   $mac,        ";",
                   $type,       ";",
                   $nb_pkt,     ";",
                   $timestamp,  ";",
                   $new_x,      ";",
                   $new_y,      ";",
                   $new_z,      ";",
                   $end_line
            );
        }
        # If the current line has at least one of its coordinates to a
        # non-default value, just print it
        else {
            output($cur_line);
        }

        # In either case, jump to the next line
        next;
    }


    # No previous line in memory: we might have got a chance to get rid of the
    # current line right now
    if (@prev_lines == 0) {
        # The current line is "complete": print it and store the coordinates for
        # later interpolation
        if (is_complete($x, $y, $z)) {
            # Print current line
            output($cur_line);
            # Store current values
            $last_x = $x;
            $last_y = $y;
            $last_z = $z;
            # Store current timestamps
            $last_x_t = $last_y_t = $last_z_t = $timestamp;
            # Jump to next line
            next;
        }

        # The current line is fully incomplete (all the coordinates at 0) and we
        # don't have previous coordinates to interpolate: this should not happen
        # if the input file is correct, so we print a warning but don't store
        # the line since we won't be able to interpolate anything anyway
        if (is_blank($x, $y, $z) && is_blank($last_x, $last_y, $last_z)) {
            print_warning("No previous X, Y and Z values to interpolate!");
            output($cur_line);
            next;
        }
    }


    # OK, the serious work starts here. The current line is either complete or
    # incomplete. If it is incomplete, it can still contain non-zero values, in
    # which case we will have to interpolate the previously stored values with
    # this value. For instance, if the current line have X = 0, Y = 42, Z = 0,
    # it is incomplete because X and Z are zero, but we will use the Y value
    # (42) as the end bound to interpolate Y values of the previous lines we
    # stored (assuming we have a begin bound value for Z). In this case, the
    # interpolation is done by modifying the lines in memory (which can still
    # have missing values even after the interpolation, but that will be checked
    # by the code above, at the next loop).
    #
    # In any case, if the line is incomplete, we will store it for later
    # interpolation (except if there are no real coordinates before this line,
    # but this should not happen if your input file is correct).
    #
    # If it is complete, we will also store it after having interpolated the
    # previous lines in memory, and all the lines will be printed at the next
    # loop in the check above (or after the end of the loop if it's the last
    # line).


    # Check if we have to compute X's value
    if ($x eq BASE_COORD) {
        if (!defined($last_x)) {
            # We've got a problem if we don't have a previous value to
            # interpolate
            print_warning("No previous X value to interpolate!");
        }
        else {
            $xs++;
        }
    }

    # Check if we have to compute Y's value
    if ($y eq BASE_COORD) {
        if (!defined($last_y)) {
            print_warning("No previous Y value to interpolate!");
        }
        else {
            $ys++;
        }
    }

    # Check if we have to compute Z's value
    if ($z eq BASE_COORD) {
        if (!defined($last_z)) {
            print_warning("No previous Z value to interpolate!");
        }
        else {
            $zs++;
        }
    }

    # Interpolate what we can if needed
    if (@prev_lines > 0) {
        # X interpolation
        if ($xs > 0 && $x ne BASE_COORD) {
            apply_interpolation($xs, X_FIELD_INDEX, $last_x, $last_x_t, $x,
                                $timestamp);
            $xs = 0;    # There are no more Xs waiting
        }

        # Y interpolation
        if ($ys > 0 && $y ne BASE_COORD) {
            apply_interpolation($ys, Y_FIELD_INDEX, $last_y, $last_y_t, $y,
                                $timestamp);
            $ys = 0;    # There are no more Ys waiting
        }

        # Z interpolation
        if ($zs > 0 && $z ne BASE_COORD) {
            apply_interpolation($zs, Z_FIELD_INDEX, $last_z, $last_z_t, $z,
                                $timestamp);
            $zs = 0;    # There are no more Zs waiting
        }
    }

    # Store the current line; if it is complete, previous lines should be
    # complete as well now, since we just interpolated what needed to do, so all
    # the lines in @prev_lines will be printed at the check at the beginning of
    # the next loop
    trace("Line stored.\n");
    push @prev_lines, $cur_line;

    # Store the current coordinates if they are non-zero
    if ($x ne BASE_COORD) {
        $last_x   = $x;
        $last_x_t = $timestamp;
    }
    if ($y ne BASE_COORD) {
        $last_y   = $y;
        $last_y_t = $timestamp;
    }
    if ($z ne BASE_COORD) {
        $last_z   = $z;
        $last_z_t = $timestamp;
    }
}


## Display the lines remaining in memory if any ##

if (@prev_lines) {
    trace("Printing " . scalar(@prev_lines) . " lines left in memory...\n");
    output($_) foreach (@prev_lines);
}
trace("The end.\n");


## Write the output file if needed ##

if (defined($output_file)) {
    # Close the output handle
    close $output_handle
      or die "Can't close the output file handle: $!";

    # Backup the original file
    my $newname = $output_file . $backup_suffix;
    move($output_file, $newname)
      or die "Can't move \"$output_file\" to \"$newname\": $!";

    # Move the temporary file to the new file
    move($tmp_output_file, $output_file)
      or die "Can't move the temporary file to \"$output_file\": $!";
}


# vim: tabstop=4:shiftwidth=4:expandtab:textwidth=80