3.2.3 Registering Stream Interfaces
Stream interfaces encapsulate access to stream data on different communi-cation and storage media providing a unified set of methods to the rest of the system. A new stream type can use some of the generic stream interfaces already available in the system. For example, the stream interfaces for inter-GSDM streams, local streams, and standard output streams are generic and can be used for all new stream types without changes. New stream interfaces can also be registered for a stream type by the means of a system procedure, register_stream_interface, with the following signature:
register_stream_interface(Charstring tpname, Charstring intname,
Charstring openfn, Charstring nextfn, Charstring insertfn, Charstring closefn) -> Boolean;
The first parameter tpname specifies the stream type for which the interface can be used and the second, intname, is the name of the interface. The last four parameters are names of the functions implementing the stream interface methods open, next, insert, and close. For example, the following call registers an interface named RadioUDP for the Radio stream type using UDP commu-nication protocol:
register_stream_interface("Radio","RadioUDP",
"openUDP", "nextUDP",
"insertUDP","closeUDP");
When a new stream source is registered to the system, the value of its inter-faceattribute is used to determine what interface implementation to be used for it. In this way different stream interfaces are supported for the same stream type and can be used by different stream objects of this type.
3.3.1 Defining Stream Query Functions
An SQF is defined on one or more input streams and can also have other, non-stream parameters. There are two kinds of SQFs: transforming SQFs that transform a single input stream, and combining SQFs combining multiple in-put streams.
The transforming SQFs have as a first parameter the input stream on which they operate. The specification contains calls to window functions to extract current logical windows from the input stream and constructors to create new logical windows. All attribute functions returning components of logical win-dows can be used in the definition. Furthermore, the SQF specification may contain any built-in or user-defined operations on attributes or entire logical windows. User-defined operations are implemented as foreign functions in, e.g., C or Java, and plugged into the system. The select clause must return logical windows of the result stream type.
The SQF fft3 below is defined on Radio stream type and computes the Fast Fourier Transform (FFT) on each of the three channels of the current logical window of the radio stream:
create function fft3(Radio s) -> RadioWindow as select radioWindow(ts(v),fft(x(v)),
fft(y(v)),fft(z(v))) from RadioWindow v
where v = currentWindow(s);
The from clause defines v to be of type RadioWindow. The where clause binds v to the current logical window of the stream s returned by the current-Windowfunction. The fft function is a foreign function that computes the FFT over a vector of complex numbers2. The function has the following signature:
fft(Vector of Complex x) -> Vector of Complex y The function is called for each of the signal channels and a result logical win-dow is created by the system generated constructor radioWinwin-dow called in the selectclause.
3.3.2 SQF Discussion
The SQF specifications are asymmetric in sense that they are defined on streams but produce a window, not a stream. If an SQF were to return a stream object, the insert method of the stream interface must be called in the select clause,
2We chose FFT as an illustration example since it is commonly used in signal processing appli-cations and is computationally expensive.
which would lead to a side effect in SQFs and violate the referential trans-parency. Therefore, we leave the insertion of the result logical window(s) into the output stream to the query executor. The select clause can construct a single logical window, a bag of logical windows, or sequences of logical win-dows.
The SQFs have to be defined over streams rather than logical windows in order to allow for aggregation of multiple subsequent data items in the input streams, as it is done by slidingWindow or timeWindow functions.
3.3.3 Transforming SQFs
The transforming SQFs provide functionality for streams similar to the func-tionality of the unary relational algebra operators project and select. In addi-tion, aggregation and arbitrary transformations can be performed. To demon-strate this, next we present some more examples of SQFs used in the radio signal application.
The SQF xchannel extracts only the time stamp and the x channel of a radio signal stream analogous to the project operator in the relational algebra. It calls a channelWindow constructor to construct a new logical window out of the two selected components from the logical window of the input stream:
create function xchannel(Radio s) -> ChannelWindow as select channelWindow(ts(v),x(v))
from RadioWindow v
where v=currentWindow(s);
A simple sampling on the stream extracting randomly one logical window out of a sequence of n is specified by the following sample function, defined on the stream type:
create function sample(Stream s, Integer n)-> Window as select w[rand(0,n-1)]
from Vector of Window w
where w = slidingWindow(s,n,n);
This function can provide functionality of load shedding by random dropping of stream items similar to the drop operator in [2]. More complex sampling algorithms implemented as functions can be plugged into the system and used by substituting the expression in the select clause.
Partitioning of streams for the purposes of parallel execution of SQFs is im-plemented as a set of partitioning SQFs, each extracting the logical windows for each partition. For example Round Robin partitioning is implemented through an SQF RRpart that specifies a single Round Robin partition on any stream s of logical windows:
create function RRpart(Stream s,
Integer ptot, Integer pno) -> Window as select w[pno]
from Vector of Window w
where w = slidingWindow(s,ptot,ptot);
The function is parameterized on the order number pno of the partition (start-ing with 0) and the total number of partitions ptot.
Selecting a sub-stream of elements fulfilling a predicate is illustrated in the next SQF. The result stream contains only logical windows of Radio source that have maximum magnitude of the x channel bigger than some threshold thrthat is a parameter of the function:
create function xmagn(Radio s, Real thr) -> RadioWindow
as select v
from RadioWindow v
where v = currentWindow(s) and max_magn(fft(x(v)))>thr;
The max_magn is a function computing the maximum magnitude of a signal applied on the x-channel of the radio stream.
The following function polar computes the parameters of the polarization ellipse of the Radio signal that characterizes the direction of the electromag-netic field. The example illustrates how the user can encapsulate several ap-plication operations on the stream data in one SQF:
create function polar(Radio s) -> PolarWindow as select polarWindow(polarization(ts(v),
fft(x(v)),fft(y(v)),fft(z(v)))) from RadioWindow v
where v = currentWindow(s);
Here polarization is a function computing six vectors of real numbers that characterize the polarization ellipses for a range of frequencies. The polar SQF returns logical windows of type PolarWindow that contain a timestamp and six vectors of real numbers.
Aggregations of stream data can also be expressed as SQFs that call appro-priate aggregate operations over the logical windows. Let aggregate_window be a used-defined function that aggregates n RadioWindows with signature:
aggregate_window(Vector of RadioWindow w) -> RadioWindow
In order to aggregate the stream using jumping, i.e. non-overlapping win-dows of size n, we specify the following SQF:
create function agg_stream(Radio s,Integer n) -> RadioWindow
as select aggregate_window(slidingWindow(s,n,n));
The same aggregation can be performed over sliding overlapping windows of the stream by adding a parameter for the overlapping step st:
create function agg_stream_sliding(Radio s, Integer n, Integer st) -> RadioWindow
as select aggregate_window(slidingWindow(s,n,st));
3.3.4 Combining SQFs
Combining SQFs are defined over more than one input stream and perform some kind of union or join to combine the stream data. To allow definitions over any number of input streams, the first parameter of combining SQFs is of type Vector of Stream. For the purposes of partitioned parallel execution of SQFs we define two combining SQFs, S-Merge and OS-Join. Other functions using different combining patterns can be defined to meet the requirements of the application.
S-Mergemerges a number of input streams ordered on an ordering attribute into a single ordered stream. The ordering attribute in our implementation is the distinguished time stamp attribute, which all timestamped streams have. S-Mergeis analogous to the relational union operator and in addition it preserves the stream order.
Since S-Merge compares logical windows from multiple streams, it may block if a stream stales and its buffer is empty. In order to provide non-blocking behavior S-Merge has a time-out parameter. The time-out is the time the system waits for a logical window from a stream if the stream buffer is empty, before assuming that the window is lost. The function signature is as follows:
S-Merge(Vector of Stream s, Real timeout) -> Window The OS-Join SQF is an analogue to the relational join operator on the stream ordering attribute, which in addition allows a function-parameter to be spec-ified that combines the logical windows with the same value of the ordering attribute. The function-parameter must be window transforming function, or in other words it must be defined on logical windows and return logical windows.
The function signature is as follows:
OS-Join(Vector of Stream s, Function combinewin) ->Window;
where combinewin has the signature:
combinewin(Vector of Window v) -> Window;