Monitoring and Managing the Servers

From Multiverse

NOTE: JMX management and monitoring is an early access feature. It is not currently complete.

Enabling management and monitoring

The Multiverse servers are configured to provide for management and monitoring using Java Management Extensions (JMX). To enable JMX management and monitoring, edit the server start script and change the set ENABLE_MGMT property to true. How you do this depends on your operating system.

You can also set the ENABLE_MGMT environment variable instead of editing the startup script. If you're using multiverse.sh then the -M option will enable JMX.

Then start the servers as usual.

Windows

On Windows, edit start-multiverse.bat and change the following line:

set ENABLE_MGMT=false

to:

set ENABLE_MGMT=true

Linux

On Linux or Unix, edit multiverse.sh and change the following line:

ENABLE_MGMT=false

to:

ENABLE_MGMT=true

Or pass the -M option to multiverse.sh

Starting the management console

You can use the jconsole tool to monitor memory consumption, class loading and thread management. The jconsole tool is part of the JDK installation (it is not included in the JRE).

After the servers have successfully started, open a shell window (either a Linux/Unix shell or a Windows command prompt window). Enter the following command to start the jconsole management and monitoring console:

jconsole

If you get the following error:

• Windows:

'jconsole' is not recognized as an internal or external command,
operable program or batch file.

• Linux/Unix:

jconsole: No such file or directory

Then, you need to add JAVA_HOME/bin to your PATH environment variable.

On Windows, open Control Panel > System. Open the Advanced tab, and click Environment Variables. Under System Variables, edit the Path variable and add the JAVA_HOME/bin directory, where JAVA_HOME is the directory where Java is installed, for example C:\Program Files\Java\jdk1.5.0_08\bin.

On Linux/Unix, add the following command to your .bashrc (or .cshrc or equivalent):

export PATH=$JAVA_HOME/bin:$PATH

Make sure that the $JAVA_HOME environment variable is also set to the directory where Java is installed.

Running JConsole

After you successfully start jconsole, you will see the connection dialog box. The dialog is different between Java 5 and Java 6.

JConsole (Java 5)

JConsole (Java 6)

This dialog box lists all the local running Java processes that are instrumented for JMX. The screenshot above shows the list of all nine Multiverse servers, along with their command line arguments. With the introduction of Multiverse 1.1, distinguishing processes by command-line argument has become more difficult. The connect dialog in JConsole 6 makes it even harder by removing the horizontal scroll-bar. You can determine the process ids of Multiverse proecsses by the .pid files in MV_HOME/bin/run/world/. An easy way to print all the pids:

grep . MV_HOME/bin/run/world/*

Click on any of the servers to monitor it. By default, Java SE provides monitoring and management of the Java virtual machine. For example, you can monitor memory consumption, by clicking on the Memory tab:



Future releases will provide additional management and monitoring capabilities.

Multiverse MBeans

Every Multiverse process contains a custom Engine MBean that reports basic information. In addition, the ProxyPlugin has an MBean for player information. The MBean names are:

• net.multiverse:type=Engine
• net.multiverse:type=Proxy

Both MBeans include some writable parameters to control the running process.

Example of Proxy MBean showing graph of CurrentUsers (double-click to get graph).

Python Scripting

The Engine MBean includes an operation to run a Python script. The operation captures and returns the script stdout and stderr (that is, anything the script prints.) The script is run in a different Python local variable scope than the world scripts, so variables from the python world scripts are not immediately available. This protects the world data from accidental changes or overwrites due to management scripts.

The Engine MBean runPythonScript() operation can be run from jconsole or the mvm command-line program. In jconsole, navigate to the net.multiverse:type=Engine mbean and select "Operations". You can enter python script in the text box and click "runPythonScript". The results will display in a pop-up window. This works fine for short one-line scripts, but does not scale to script files.

mvm

You can use the mvm command-line program to run a python script inside any Multiverse server process (except the domain server). The script can be in a file or on the command-line. Extra arguments to mvm are passed along to the script in an argv array.

Usage: mvm -p <pid|agent-name> -s <script> -f <script-file-name> [...]

The mvm program, located at MV_HOME/bin/mvm, is a bash shell script. On Windows, Cygwin is required to use mvm. The options are:

-p <pid|agent-name>

Process identifier, either a process id or Multiverse agent name. Use of the agent-name is only available with the Sun JDK.

-s <script>

Run the <script> and print the results.

-f <script-file-name>

Run the contents of <script-file-name> and print the results.

[...]

Any additional arguments are available to the script in the argv array. The first argument is at array index 1. To pass a script argument with the same option as an mvm option, use --: mvm -p objmgr -f myscript.py -- -p script-argument

The agent-name is the same as the process logger name (set on the server command line via <code>-Dmultiverse.loggername). The agent name is available as an Engine MBean attribute and can be seen in the process log file. Java process ids can be found using jps. Running jps -v will display the logger name, so you can identify agents. If you are running multiple worlds on the same machine, at the same time, then multiple processes will have the same agent name.

The argv array is set by prefixing one line to the beginning of the script. This causes line numbers in python errors to be off by one.

The following module imports are performed on the Python environment:

import sys
from multiverse.mars import *
from multiverse.mars.objects import *
from multiverse.mars.core import *
from multiverse.mars.events import *
from multiverse.mars.util import *
from multiverse.mars.plugins import *
from multiverse.msgsys import *
from multiverse.server.plugins import *
from multiverse.server.math import *
from multiverse.server.events import *
from multiverse.server.objects import *
from multiverse.server.worldmgr import *
from multiverse.server.engine import *

If your script publishes messages, then you'll need to run it in a process that already advertises the message types. Note that all *Client functions publish messages (e.g. WorldManagerClient, InstanceClient).

To see the arguments received by python:

mvm -p objmgr -s "print argv" arg1 arg2

this will print:

['<string>', 'arg1', 'arg2']

The following example scripts can be saved to files and used with -f.

Create an instance: mvm -p instance -f createInstance.py <template-name> <instance-name>

# createInstance.py
ii = 1
persistent = None
while ii < len(argv):
    arg = argv[ii]
    if arg == "-P":
        persistent = Boolean(True)
    elif len(arg) > 0 and arg[0] != '-':
        break
    else:
        print "Unknown option " + arg
    ii = ii + 1

templateName = argv[ii]
instanceName = argv[ii+1]

print "persistent "+str(persistent)
print "templateName "+templateName
print "instanceName "+instanceName

overrideTemplate = Template("instance");
overrideTemplate.put(Namespace.INSTANCE,
    InstanceClient.TEMPL_INSTANCE_NAME, instanceName)
if persistent != None:
    overrideTemplate.put(Namespace.OBJECT_MANAGER,
        ObjectManagerClient.TEMPL_PERSISTENT, persistent)

rc = InstanceClient.createInstance(templateName, overrideTemplate);
print "createInstance oid=" + str(rc)

import sun.management.ConnectorAddressLink;
import javax.management.remote.JMXServiceURL;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.JMException;

   static void runRemoteScript(int processId, String script)
   {
       try {
           JMXConnector jmxc;
           MBeanServerConnection server;

           String address = ConnectorAddressLink.importFrom(processId);
           JMXServiceURL jmxUrl = new JMXServiceURL(address);
           jmxc = JMXConnectorFactory.connect(jmxUrl);
           server = jmxc.getMBeanServerConnection();
           Object[] parameters = {script};
           String[] signature = {"java.lang.String"};
           Object result;
           result = server.invoke(new ObjectName("net.multiverse:type=Engine"),
               "runPythonScript", parameters, signature);
           System.out.println(result.toString());
           jmxc.close();
       }
       catch (IOException e) {
           System.err.println("Unable to attach to " + processId +
               ": " + e.getMessage());
           return;
       }
       catch (javax.management.MalformedObjectNameException e) {
           System.err.println("Internal error: " + e.getMessage());
           return;
       }
       catch (javax.management.InstanceNotFoundException e) {
           System.err.println("Process "+processId+" is not a Multiverse engine");
           return;
       }
       catch (javax.management.MBeanException e) {
           System.err.println("Error: "+e);
           return;
       }
       catch (javax.management.ReflectionException e) {
           System.err.println("Error: "+e);
           return;
       }
   }

To set Java command line options for the Multiverse servers, edit the <code>multiverse.sh file, and change the line

JAVA_FLAGS="-Dmultiverse.msgsvr_hostname=...

to:

JAVA_FLAGS="-option1 -option2 ... -Dmultiverse.msgsvr_hostname=...

For example:

JAVA_FLAGS="-Xmx128m -Dmultiverse.msgsvr_hostname=...

Key JVM options

For general information on JVM command-line flags, see Command Line Options - Java Application Launcher.

Specifing Heap size

Two options control the size of heap memory allocated:

• -Xms: Specify the initial size, of the memory allocation pool. Use "K" to specify KB, and "M" to specify MB. The default value is 2MB.
• -Xmx: Specify the maximum size, in bytes, of the memory allocation pool. Use "K" to specify KB, and "M" to specify MB. The default value is 64MB.

For example, to manually set the maximum heap space to 128MB, use this option: -Xmx128m.

If you encounter the error:

java.lang.OutOfMemoryError: Java heap space 

Then increase the maximum memory allocation.

Specifying client or server virtual machine

J2SE 5.0 has two distinct VMs:

• client VM tuned to minimize application start-up time and memory footprint
• server VM tuned to maximize peak operating speed at the expense of startup time and memory consumption.

The JVM automatically detects a "server class" machine (one with at least 2 CPUs and at least 2GB of physical memory), and uses the server VM when appropriate, as descrbed in Server-Class Machine Detection.

By default, the server startup scripts use the server Java VM.

For development, in which you are restarting the servers often, the client VM may be appropriate, even if you are using "server class" hardware. Use the -client command line option to specify the client VM.

For testing and production, when the servers are expected to run for many hours or days continuously, the server VM is most appropriate, even if your hardware is not strictly "server class." Use the -server command line option to specify the server VM.