Figure 3.1: Vendetta layout for a DHT testbed
The node client
The node client runs at all nodes in the distributed testbed and can be started from the monitor with a SSH connection.It acts as a middle-man between the testbed components running at the node and the monitor. The applications are started as processes and the process output is parsed by the node client. A benefit of this is that applications can be evaluated even if they are closed source, but in that case, the resolution of the monitoring is limited to the amount of log information the application outputs. The node client collects log data from the testbed and stores it in a local log.
Requested log data are sent to the monitor for presentation. It is possible for the monitor to instruct the node client to filter its log data before sending it to the monitor. However, the node client still stores all log events locally so that it can be used to do post-mortem analysis of the networks behavior. To support management tasks, the node client can also receive messages from the monitor, for example messages telling it to start or stop the monitored application.
It is important to note that the node client does not make any assump-tions about the testbed that it serves. Instead, it act as a kind of message-passing middle-man between the nodes in the testbed and the monitor. A benefit of having a middle-man software is that if the monitored application dies during an experiment, the node client can restart it and notify the the monitor. How data is collected, filtered and communicated is defined in a configuration file read by the node client when it starts up. By specifying another configuration file, the node client can be reconfigured to interact with another type of testbed. In section 3.3, we present an example of what a configuration file for the node client can look like in a specific testbed.
The monitor
The monitor is the program that interacts with the testbed and present information to the user. The GUI is divided into three areas. First is the node list that is read from a node file at startup. The node list contains all nodes that is part of the testbed that you want to monitor. The second area is for commands that you can send to one or many nodes. Typically the commands are implemented as buttons that might have a form for extra arguments. However some commands are chosen with drop down menus, which is very effective when you know that the possible options are limited.
The third area is the canvas area. This area uses most of the screen to visualize the events received from node clients. The number of, and which, graphical canvases displayed in the canvas area is defined in the monitor
configuration file. As canvases are dynamically loaded at runtime, it is not necessary to recompile Vendetta to support a new canvas.
When the monitor is initiated, it waits for node clients to report . As nodes clients are heard from, their status is updated in the GUI and the user can start to interact with them. If the node client software is not running on the remote nodes, the user can use the GUI to initiate the client software. Nodes report to the monitor by sending alive messages. The alive messages, or rather the lack of them, are also used to determine if a node is unavailable. Because the monitor waits for alive messages and node clients are not expecting the monitor to respond, the monitor is not a single point of failure. If the monitor would crash or be closed down, e.g. during longer tests, the user only needs to restart the monitor to regain control of the testbed.
To reduce the amount of data sent over the network, the monitor can instruct node clients to use a specific filter. It can also request all logged data from one or more node clients to get a more complete picture. The user can decide whether a certain type of log event should be sent using UDP or TCP. Typically periodic node state would be sent using UDP whilst important but rare events are sent using TCP. The node client also buffers log events before sending them to reduce the number of datagrams the monitor need to handle. The buffer is either sent when full, or when a configurable amount of time has passed.
Events can be monitored in several different ways. Live monitoring gives an overview of what is currently going on in the testbed. As all received data is saved, it is also possible to pause the live monitoring, rewind and replay interesting sequences. When not monitoring events live, the visual-ization speed can be changed to either fast forward past uninteresting event sequences, or slowly replay the interesting ones.
There are two ways to send commands from the monitor to the node client, either as a pre-defined command that is parsed by a Java class on the receiving node, or as SSH commands. The Java commands are in clear text and not tested for origin, therefor we do not accept anything other than requests of predefined data units. A button can trigger shell commands over SSH if that flexibility is needed. If a user needs complete control over a remote node, she can start a terminal over SSH to the remote node client by clicking on the selected node and request a terminal. This is substantially easier than logging in on the remote node in a more conventional way.
A playout buffer is used when monitoring live. The length of the play-out buffer is configurable, rather than adaptive. Having an adaptive playplay-out buffer might improve the live experience but has not been prioritized as we believe the replay function is what primarily will be used for analysis
pur-Figure 3.2: Canvas showing location of nodes and selected testbed traffic
poses.Beside the monitoring features, the monitor can also perform mainte-nance tasks. Examples include starting up and shutting down testbed nodes, executing remote commands at several nodes, sending command scripts to choreograph the behavior of nodes, and more. Most management tasks ap-pear as clickable command buttons.
A key principle in the design of Vendetta is flexibility. It is possible to configure both the monitor and node clients to fit into almost any distributed testbed where the participating nodes can run Java programs. In fact, the entire behavior of Vendetta is defined in two configuration files - one for the monitor and one for the node clients. At the monitor, the configuration file defines what elements to display at the screen, what actions to associate with GUI elements such as buttons, and what nodes to communicate with. In the node client, the configuration file defines how to collect and parse data to be sent to the monitor. In section 3.3, we will present more details on how we have used Vendetta to interact with a experimental DHT deployment and what the corresponding configuration files might look like. This will give an indication of the flexibility offered.
3.2.1 Vendetta and PlanetLab
Not all behavior can be defined in configuration files. For some purposes, it may be needed to write a Java class that performs some specialized task.
One example of this is the PlanetLab module included in Vendetta.
PlanetLab[2] is a distributed testbed consisting of several hundred nodes
(a) Query path (b) Routing table and key distribu-tion
Figure 3.3: Application specific canvas illustrating a DHT ring
around the world, available through ssh remote login. Running experiments on the PlanetLab include keeping track of what nodes are active, deploying compiled code to selected nodes, coordinating node actions, collecting results and more. Throughout the years, users of the PlanetLab have developed and redeveloped scripts and tools for these tasks.
Motivated with the popularity of the PlanetLab, Vendetta includes mod-ules for monitoring and management of testbeds based on PlanetLab. For visualization purposes, a canvas showing the geographic locations of available nodes can be used to show node events. By associating an event at a node with a graphical representation, it becomes possible to create more intuitive ways to parse data. As an example illustrated in figure 3.2, query paths in a distributed system can be visualized on the globe, making it obvious when inefficiencies occur due to long communication paths.
For management purposes, Vendetta includes a module that ease code deployment on the Planet-lab. Nodes to monitor or manage are selected from a list or by clicking on the canvas. By clicking on a button in the GUI, a custom command can be executed at selected nodes. Filters to use, graphical canvas associations, commands to execute and events to monitor are all specified in the configuration file that defines the testbed behavior.