Server Scalability Testing
From Multiverse
| Multiverse Servers |
|
Installing • Installing on Linux • Running • Troubleshooting • FAQ • Release Notes • Updating • JMX Monitoring & Mgmt. |
| Infrastructure |
|
Platform Architecture • Registering a World • Proxy Server • Event Handling • World Manager • Voice Server |
| Messaging System |
|
Perception Messaging • Using Extension Messages • Message Marshalling • Multi-subject Messaging • Message Catalog |
| Object Architecture |
|
World Instancing • Server Object Search • Server Regions • Server Markers |
| Scalability and Performance |
|
Overview • Scalability Testing |
| Reference |
|
File Layout • Property File • Logging • API |
Contents |
Overview
Multiverse performs scalability testing with a test client (also called a "bot") that simulates a real client's activities. From the server's point of view, test clients are just like player-operated clients, because the server can't anticipate their actions.
Scripts
Two scripts are used to perform load testing:
-
mv-home/bin/simpleclient.sh -
mv-home/bin/multisimple.sh
Running the test client
The servers come with a simple test client to enable you to check that your servers are running properly and a local client can log in. You can also use the test client for scalability testing.
Command syntax
Use the mv-home/bin/simpleclient.sh script to run the test client.
./simpleclient.sh
Running multiple test clients
Use the mv-home/bin/multisimple.sh script to run multiple test clients simultaneously for load testing. The command syntax is:
./multisimple [numClients]
Where numClients is the number of test clients to run. The default is ten (10).
Running a single test client
The server start script multiverse.sh includes a "test" command to run a single test client (if you are on Linux or Windows with Cygwin), as follows:
./multiverse.sh test
You can also run the test client using the full command-line syntax:
java multiverse.simpleclient.SimpleClient [-e propertyFile] -s $MV_COMMON/simpleclient.py --exit-after-login
where propertyFile specifies the property file to use. If you don't specify a property file, the test client uses the default, multiverse/config/common/simpleclient.props.
See Command Line Syntax for a general description of command-line syntax notation.
Properties file
The default properties file is mv-home/config/common/simpleclient.props. It specifies some key properties for the test client. Use the "-e" command option to specify a different properties file.
use_master_server=false # Use this if use_master_server is false account_id=1 # Need these only if use_master_server is true #username #password # Need these only if use_master_server is true #ms_tcp_port (int) #ms_rdp_port (int) world_id=sampleworld login_hostname_override=localhost login_port_override=5040 log_level=1
Normally, if you are testing connection to your servers, you should keep the default, use_master_server=false, so the client will connect to the world specified with the properties:
- world_id: name of the world
- login_hostname_override: hostname of the world server
- login_port_override: port number of the world sever
Configuration script
The Python script mv-home/config/common/simpleclient.py specifies some other configuration parameters. Use the -s command line option to specify a different configuration script.
The LoginResponseHandler will look for a proxy login response and exit if --exit-after-login is on the command line.
loginType = 81
simpleClient = SimpleClient.getInstantiatingSimpleClient();
# LoginHandler
loginResponseHandler = LoginResponseHandler()
simpleClient.getDispatcher().registerHandler(loginType,
loginResponseHandler)
Multi-client testing
Scalability testing uses two classes:
-
multiverse.simpleclient.SimpleClient- provides a light-weight process that handles the login protocol as well as client position interpolation -
multiverse.simpleclient.PlayerClient- introduces "behavior" by giving the test client avatar direction and orientation.
We run many instances of the PlayerClient/SimpleClient test client on one machine, and the Multiverse server on a different machine. In addition to the test client instances, which appear to the server as real clients, we log in using real clients, to watch events in the world.
Increasing the server's open file limit
With large number of client connections, the server may exceed the operating system's per-process maximum number of open files. This is only needed for machines running the ProxyPlugin.
Linux
To view the current file descriptor limit (bash):
ulimit -Hn
Increasing this limit requires modifying /etc/security/limits.conf (root access required). Add the following line to increase the hard limit to 8192 for all users:
* hard nofile 8192
This limit will take effect for subsequent logins. Existing processes are unaffected. See limits.conf or "man limits.conf" for more configuration options.
Windows
TBD.
Testing notes
Multiverse performed an initial set of tests with the bots simulated on one quad-core server, and the Multiverse servers running on a different quad-core server. The load per bot on the simulating machine was about 40% for all 150 bots, and on the server machine between 10 and 12 percent. We haven't worked to reduce the CPU load of the simulation code yet, but it looks like we'll have to do so soon.
The approximate per-process CPU loads measured were:
- Proxy server - 5%
- World manager - 3%
- All the other processes - 2%
We've done only a limited amount of multi-server testing, and not under enough load to stress anything. The aggregate load for all the processes wouldn't be much different if each process (proxy, world manager, mobserver, and so on) were on a different machine, so we expect that to scale up very well. The proxy server easily scales on multiple machines by hashing player OIDs to determine the proxy server to use. We believe that this will scale linearly with no surprises.
The more interesting case is multiple world manager processes which split up the terrain between them. We expect that will scale as well, though not linearly if all the bots are near the boundary between the world managers.
The simulation algorithm is simple: generate a random number to set the direction, another for how long the bot will run in that direction, and a third for how long the bot will be idle after running in that direction. So it's mostly straight line movement, but since the idle periods are pretty short, the bots are moving nearly all the time. With 150 bots, we're seeing message counts sent to each bot to keep it up to date on the directions and positions of the other bots on the order of 80 - 100 per second, or one message every 1.4 seconds on average.
The network load to each bot/client was on the order of 5KBytes/sec, because all the bots are in perceiver range of each other. A more realistic test would be to have the bots come in and out of range of each other. That setup would reduce the messaging because each bot would get messages generated by a smaller number of bots, but increase the load to handle moving the bots in and out of perceiver range of each other. My guess is that the net effect of having, say, 1/3rd of the bots in range of each other any moment is that the messaging load would be somewhat smaller, but the overall server load would be about the same.
The default perceiver range is 100 meters; for this test, to keep things simple, we chose a perceiver range of 1000 meters, which guaranteed that all bots would be in range of each other.
We haven't done any formal comparison between mobs and the test client bots. But we'd expect the load to be pretty similar, because the server manages mobs by sending them MobPath messages, and doesn't need to send any additional messages until it supplies the next MobPath message.
