Server Message Marshalling

From Multiverse

Jump to: navigation, search
Multiverse Servers

Installing  •  Installing on Linux  •  Running  •  Troubleshooting  •  FAQ  •  Release Notes  •  Updating  •  JMX Monitoring & Mgmt.

Infrastructure

Platform Architecture  •  Registering a World  •  Proxy Server  •  Event Handling  •  World Manager  •  Voice Server

Messaging System

Perception Messaging  •  Using Extension Messages  •  Message Marshalling  •  Multi-subject Messaging  •  Message Catalog

Object Architecture

World Instancing  •  Server Object Search  •  Server Regions  •  Server Markers

Scalability and Performance

Overview  •  Scalability Testing

Reference

File Layout  •  Property File  •  Logging  •  API

Contents

[edit]

Introduction

To be able to send messages over the network, the Multiverse servers must serialize or marshal the messages. Marshalling transforms the memory representation of an object to a data format suitable for storage or transmission.

Java's serialization facilities are not efficient enough to support the volume and of messages Multiverse servers require. To address this, Multiverse uses byte code injection to generate marshalling and unmarshalling code at runtime. This generated code runs fast and results in compact serializable representations of messages.

When writing classes that will be marshalled, you must follow some rules to ensure that class definitions are compatible with generated marshalling. Also, your code must conform to some constraints due to the server's use of a custom Java class loader.

[edit]

Overview of marshalling

All server processes must:

  • Agree on the wire format used for messages
  • Be able to enumerate the set of classes that will be marshalled
  • Must generate the marshalling for those classes before any other use of the classes.

The class multiverse.server.marshalling.MarshallingRuntime provides the APIs to enumerate the classes to be marshalled, and the runtime routines that generated marshalling code invokes.

The interface multiverse.server.marshalling.Marshallable contains two methods:

  • marshalObject() - marshals an object into the byte buffer argument.
  • unmarshalObject() - Unmarshals an object from the byte buffer argument, returning an object containing the unmarshalled state. Nearly all implementors of Marshallable unmarshal the state into this, and return this. However, some implementors need to "intern" the result of unmarshalling, and will potentially return a value different from this. This provides the functionality of the java.io.Serializable interface readResolve() method. Examples in the server of classes that intern objects during unmarshalling include MessageType and Namespace.

The method signatures are: 

public void marshalObject(MVByteBuffer buf);

public Object unmarshalObject(MVByteBuffer buf);

The virtual function marshalObject takes an MVByteBuffer argument, and marshals this into the byte buffer, calling the marshalObject methods on it's supertype, if any, and on those data members that are not primitive types, and makes direct calls to into the marshalling runtime marshal primitive and built-in types. The virtual function unmarshalObject performs the inverse function, returning the object unmarshalled. When you call unmarshalObject by hand, you must create an instance of the type to be unmarshalled, typically by calling the no-argument constructor, in order to invoke the unmarshal method. The object returned may be a different object than was passed in; this is useful in cases where the semantics of the unmarshalled object require it to be interned, and provides the same functionality that Java serialization provides via the readResolve() method. The server classes MessageType and Namespace make use of this intern capability.

Nearly all marshallable classes have their marshalling code automatically generated at runtime. However, in some cases, you must explicitly define marshalling code. For example, there are several classes that explicitly marshal code to transform the instance state in the process of putting it on the wire. A class that defines explicit marshalling must implement Marshallable, defining marshalObject and ummarshalObject methods. Internally, the explicit marshalObject method will call MVByteBuffer.putxxx methods to marshal primitive types, and static methods on MarshallingRuntime to marshal user-defined classes and aggregate types such as List, Map, Set and ArrayList. Similarly, the explicit unmarshalObject method will call MVByteBuffer.getxxx methods to unmarshal primitive types and static methods on MarshallingRutime methods to unmarshal user-defined classes aggregate types such as LinkedList, HashMap, HashSet and ArrayList. MarshallingRuntime.marshalObject() adds a one- or two-byte type code for each object marshalled, so that the unmarshalling methods can determine how the data should be unmarshalled.

[edit]

API documentation

The package multiverse.server.marshalling provides complete Javadoc API documentation.

[edit]

Class registration

The MarshallingRuntime has compile-time known type code assignments for primitive types (boolean, byte, double, float, int, long, short, String), as well as aggregate types (LinkedList, HashMap, HashSet, ArrayList and array of byte). All other classes must be registered so that type codes can be assigned, and marshalling methods can be injected at startup.

All multiverse classes to be marshalled are registered in the file multiverse/config/common/mvmarshallers.txt. The first few lines of that file are: 

# server.engine marshalled objects
     multiverse.server.engine.BaseBehavior$ArrivedEventMessage
     multiverse.server.engine.BaseBehavior$FollowCommandMessage
     multiverse.server.engine.BaseBehavior$GotoCommandMessage
     multiverse.server.engine.BaseBehavior$StopCommandMessage
     multiverse.server.engine.BasicWorldNode
     ...

There are about 230 classes registered for marshalling in the current Multiverse server.

Register world-specific classes to be passed in (or as) messages in the file multiverse/config/worldname/worldmarshallers.txt.

[edit]

Rules for marshallable classes

You must observe some rules when writing classes for which marshalling code is automatically generated at startup. The servers check nearly all of these rules at startup. In a rule is violated, the server processes won't start.

The rules are:

  • The class must be public.
  • The class must have a public constructor that takes no argument.
  • If the class is a nested class, it must be declared public static. Note: This is the only rule that is not checked at server startup, because the byte code injection library we use doesn't accurately report non-static nested classes.
  • All classes referred to by class state, including the supertype and any user-defined classes used as data members, must be registered and marshallable.
  • A Java aggregate type (LinkedList, HashMap, HashSet and ArrayList) cannot be a super-type.
  • The injected byte code knows the relationship between List and LinkedList, Map and HashMap, and so on. It doesn't have any way to know what a Collection is at byte-code injection time, so marshalling of Collections will always call marshalObject().
  • Just as in the case of Java serialization, data members declared static or transient will not have marshalling generated for them.
  • The class may not have a Java enum as a data member. The marshalling runtime won't allow the server to start up if it encouners a data member of type enum in a class registered for byte-code injection. Note: We may remove this restriction in the future.
  • The class may not have a Java array as a data member, unless it is an array of byte. The marshalling runtime won't allow the server to start up if it encounters a data member of type array in a class registered for byte-code injection. Note: We may remove this restriction in the future.
  • In the interest of speed, the generated marshalling code will not preserve sharing relationships. That is, if you marshal a class that has two object-valued data members that happen to point to the same object, when it is unmarshalled, you will have two copies of the object.

During initialization MarshallingRuntime analyzes the lattice of registered types to ensure that these rules are satisfied. If they are not satisfied, the server process will log the problems and exit.

[edit]

Marshalling runtime APIs

Apart from the registration APIs already described, the most interesting APIs are the static methods that marshal and unmarshal arbitrary objects: 

    public static void marshalObject(MVByteBuffer buf, Object object);
    public static Object unmarshalObject(MVByteBuffer buf);

MarshallingRuntime.marshalObject() looks up the type code of the class of the object argument in the table of class names it knows how to marshal, and outputs that type code as a byte if it's less than 256, or as a short if it's greater. For user-defined types, it casts the type to a multiverse.server.marshalling.Marshallable and invokes the object's marshalObject method. For primitive types, it calls the appropriate MVByteBuffer method ot output the type. For aggregate types, it calls the static method of MarshallingRuntime to handle the type.

Similarly, MarshallingRuntime.unmarshallObject() reads the type code from the MVByteBuffer argument. If it is the type code of a primitive type, it calls the appropriate MVByteBuffer method to read the type, boxes the result and returns it. If the type code belongs to a user-specified type, it creates an instance of the type by calling it's no-argument constructor, and then invokes the object's unmarshalObject method, returning the result. Finally, the type code specifies an aggregate type, it calls the static method of MarshallingRuntime to read the type.

For each kind of aggregate type, MarshallingRuntime has a pair of static methods to marshal and unmarshal the type. The marshaling method always takes an multiverse.server.network.MVByteBuffer and an Object as arguments (and does not return a value). The unmarshalling method always takes an multiverse.server.network.MVByteBuffer as argument, and returns an Object.

Data type marshalling method unmarshalling method
LinkedList marshalLinkedList() unmarshalLinkedList()
HashMap marshalHashMap() unmarshalHashMap()
HashSet marshalHashSet() unmarshalHashSet()
ArrayList marshalArrayList() unmarshalArrayList()
[edit]

The last resort: Java serialization

If the MarshallingRuntime does not have a registration for a class that must be marshalled or unmarshalled, it backs off to Java serialization, using a type code reserved for Java serialized object. Java serialization is slow and verbose, so it's best if it's not used often, but it does provide a backstop for cases not currently handled by the runtime. Whenever Java serialization is used to marshal or unmarshal an object, a warning message is logged (using Log.warn()). You will see these messages in the log if your log_level is 3 or less.

If you are writing explicit marshalling code for a class, and have data that you know will require Java serialization, you can call the MarshallingRuntime static methods defined for the purpose: 

    public static void marshalSerializable(MVByteBuffer buf, Object object); 
    public static Object unmarshalSerializable(MVByteBuffer buf);
[edit]

When are type codes used?

Briefly, type codes must be output in two cases:

  • The method MarshallingRuntime.marshalObject must write a type code for the object to be marshalled, because there is no a priori way for the runtime to tell what the type of the object was at the time of unmarshalling without the explicit type code.
  • When a user-defined class instance data member marshalled. The reason a type code is required is that there is no way to tell at compile time if some subclass of the data member's declared class was actually in the data member.

In contrast, no type codes are written for supertypes, or for primitive type or aggregate type data members.

[edit]

Server startup sequence

A special classloader, multiverse.server.marshalling.MarshallingClassLoader, mediates byte-code injection of marshalling code. This classloader recognizes registered classes that must have byte-code injection performed when they are loaded. The server startup script bin/multiverse.sh and all server test scripts must pass the Java argument -Djava.system.class.loader=multiverse.server.marshalling.MarshallingClassLoader on the command line, and the main Java class must be multiverse.server.marshalling.Trampoline. Trampoline initializes the MarshallingRuntime, which processes the files multiverse/config/common/mvmarshallers.txt and config/worldname/worldmarshallers.txt, if it exists, and then performs byte-code injection on all the classes registered that don't have explicit marshalling. Trampoline then starts the server process class, in most cases multiverse.server.engine.Engine.

Retrieved from "http://update.multiverse.net/wiki/index.php/Server_Message_Marshalling"
Personal tools