How to optimize wowza

This article provides suggestions for tuning Wowza Media Server on your production hardware.

Note: For use with Wowza Media Server® 2 and later.

 

  1. It is generally best to run the most recent Sun Java JDK (Java SE Downloads). It is best to run a 64-bit operating system with the 64-bit Java VM. The 64-bit VM enables Java heap sizes greater than 2GB.
    Note: If you are on a Mac with OS X 10.5.x or greater (Snow Leopard) with 64-bit hardware you can run the Java 6 VM. To change your default Java VM in OSX, open the Java preferences application /Applications/Utilities/Java/Java Preferences and drag the Java SE 6 item to the top of the Java Applications list.
  2. It is best to run the -server version of the Java VM. The -server option is enabled on Linux and Mac OS X by default. It is only on Windows where this option is not enabled by default. To enable the -server version of the Java VM on Windows, do the following:
    1. Install the latest version of the Oracle Java JDK and be sure that the JAVA_HOME environment variable is pointing to this Java installation.
    2. Edit the [install-dir]/bin/setenv.bat file to add the -server option to the startup scripts. For example:
      Code:
      set JAVA_OPTS=%JAVA_OPTS% -server
      Note: Affects both Windows Standalone and Windows Service, uncomment line 7.
  3. The default Java heap size for Windows is set to 768MB, and for Linux is set to 1200MB, which is great for development, but not large enough for high volume production use. If you are running the 64-bit version of the Java VM and have 4GB or more of RAM in your machine, we suggest a Java heap size of between 3000 MB to 5000 MB. If you have 16 GB or more RAM, we suggest a heap size of 8000 MB. We do not suggest a heap size above 10 GB as it can lead to long GC cycles/pauses. If you are running the 32-bit version of the Java VM or have less than 3GB of RAM in your machine, we suggest a Java heap size of 1500MB. Do not try to set a heap size greater than 1800MB when using the 32-bit VM. On many Linux platforms the server will start but will fail with an OutOfMemory error after a short period of time. Memory settings can be adjusted in the following script files (based on platform):
    • [install-dir]/bin/setenv.sh (Linux, Mac OS X and Solaris, around line 4)
      Code:
      JAVA_OPTS="-Xmx3000M"
    • [install-dir]/bin/setenv.bat (Windows Standalone and Windows Service, around line 4)
      Code:
      set JAVA_OPTS=-Xmx3000M
  4. Garbage collection (GC) tuning in Java is tricky. What works best in one server setup may not work in another. Through trial-and-error and customer feedback, we have several suggested approaches:Garbage Collection Options:

    We strongly recommend No Additional Settings. The default -server tuning (see step #2 above) works well for many streaming situations, so in most cases this tuning works great without additional settings.

    Only if you are experiencing problems with the Java Heap, such as a build-up of memory usage, should you try implementing one of the following Garbage Collection options:

    1. Concurrent Collector: The concurrent collector is designed for applications that prefer shorter garbage collection pauses and that can afford to share processor resources with the garbage collector while the application is running. The suggested additional settings are:If heap size is 5000 MB or greater, NewSize=512m
      If heap size is 3000 MB – 5000 MB, NewSize=256m
      If heap size is less than 3000 MB, NewSize=128m

      For example, with 64-bit OS, 8 GB RAM, and heap size of 6000 MB

      Code:
      -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m
    2. Experimental: Garbage-First garbage collector (G1): In the Sun Java 6 (Update 14 or greater) release, there is a new garbage collector that is in the experimental stage. It is designed for low pause time, high-throughput applications such as Wowza Media Server. It is an option to explore. There is more information below. To enable this garbage collector, the additional settings are:
      Code:
      -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

    To change the garbage collection settings, modify the following scripts:

    • [install-dir]/bin/setenv.sh (Linux, Mac OS X, and Solaris, uncomment line 10). For example:
      Code:
      JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m"
    • [install-dir]/bin/setenv.bat (Windows Standalone and Windows Service, uncomment line 10). For example:
      Code:
      set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:NewSize=512m

    There is more detailed information here:

    Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning
    Java SE 6 Performance White Paper
    The Garbage-First Garbage Collector (G1)

  5. For low-latency chat applications, it is best to use smaller socket buffer sizes (16000 bytes for read and write). The socket buffer sizes are configured in [install-dir]/conf/VHost.xml:
    Code:
    <ReceiveBufferSize>16000</ReceiveBufferSize>
    <SendBufferSize>16000</SendBufferSize>
  6. On Linux, increase the maximum number of file descriptors. This will address the Too many files open error message. To do this, edit the following two files:/usr/local/WowzaMediaServer/bin/wms.sh

    from:

    Code:
    #ulimit -n 20000

    to:

    Code:
    ulimit -n 20000

    /usr/local/WowzaMediaServer/bin/startup.sh

    from:

    Code:
    #ulimit -n 20000

    to:

    Code:
    ulimit -n 20000
    Note: This may fail on some versions of Linux or if you do not have permission to change the descriptor limit. If after making this change Wowza Media Server will not start, consult the documentation for your Linux distribution to see how to increase the descriptor limit.
    Note: On some versions of Linux, there is also a limit in the kernel that may need to be increased. Consult the documentation for your distribution. You may need to add the following line to your /etc/sysctl.conf file:

    Code:
    fs.file-max=20000
  7. On Linux, switch to using the Anticipatory elevator (as) elevator algorithm. It is described here:Choosing an I/O Scheduler for Red Hat
  8. On Linux, mount your disk with the noatime option. This operation will differ based on your Linux distribution. Here is a basic description of the setting:Using noatime Attribute
  9. To tune your server based on the available CPU resources of your server, use the following guidelines:The [total-core-count] refers to the total number of CPU cores in your server. For example, if you have dual quad core processors (two quad core processors), the [total-core-count] is:
    Code:
    2 (processor) x 4 (cores) = 8

    If your server supports hyper-threading, then use the total number of threads. In the above example, if hyper-threading is available, then the total number of threads is:

    Code:
    2 (processor) x 4 (cores) x 2 (threads per core) = 16

    With the number of cores and threads per physical processor continually growing, we suggest a maximum number of threads for each value below:

    In the configuration file [install-dir]/conf/VHost.xml:

    Code:
    HostPort/ProcessorCount: 2x[total-core-count] (maximum of 24) Note: The HostPort/ProcessorCount field in the Admin HostPort (/Port "8086") should not be modified.
    IdleWorkers/WorkerCount: 2x[total-core-count] (maximum of 24)
    
    NetConnections/ProcessorCount: 2x[total-core-count]  (maximum of 24)
    
    RTP/UnicastIncoming/ProcessorCount: [total-core-count]  (maximum of 12)
    RTP/UnicastOutgoing/ProcessorCount: 2x[total-core-count]  (maximum of 24)
    
    RTP/MulticastIncoming/ProcessorCount: [total-core-count]  (maximum of 12)
    RTP/MulticastOutgoing/ProcessorCount: [total-core-count]  (maximum of 12)
    
    HandlerThreadPool/PoolSize: (60x[total-core-count]) (maximum of 480)
    TransportThreadPool/PoolSize: (40x[total-core-count]) (maximum of 320)

    This assumes you have at least 1GB of memory per core and that in the case you have 4 or more total cores you are running the 64-bit Java VM using the suggested memory settings above.

  10. Running Multiple Virtual Hosts:If you are running more than one virtual host, the resource allocations must be distributed between each VHost. The simplest approach is to divide the settings listed above (which are intended for a single VHost) and distribute the resources across each VHost on your system. The settings do not have to be evenly divided, however, the total should equal what you would allocate if you were configuring for asingle VHost. If one of your VHosts will be idle most of the time, you may allocate more memory than the combined total. Be careful with this setting because excessive allocations are risky. An outOfMemory error will occur in the event both VHosts exceed the combined, available resources.

    Alternatively, in [install-dir]/conf/VHost.xml, you can set each VHost/HandlerThreadPool/PoolSize to “0” and VHost/TransportThreadPool/PoolSize to “0” which will cause the [install-dir]/conf/Server.xml settings for these properties to be used instead. This will instruct Wowza Media Server to manage the PoolSize amongst VHosts.

    A mixed approach can also be used with the [install-dir]/conf/VHost.xml file by setting the PoolSize to “0” for idle/minimal-use VHosts while using higher values for busy/high-performance VHosts with higher resource requirements.

  11. If you are not doing low latency streaming and you have a client side buffer of 3 or more seconds (NetStream.bufferTime), you can reduce the CPU load on the machine and handle more concurrent sessions by editing [install-dir]/conf/VHost.xml and changing the following two values:
    Code:
    IdleWorkers/CheckFrequency: 100
    Client/IdleFrequency: 500
  12. If you are having problems with multiple incoming multicast streams interfering with each other on Linux you may need to set the Java property java.net.preferIPv4Stack to true. To do this, edit [install-dir]/bin/setenv.sh and uncomment the following line (line 13):
    Code:
    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
  13. This really isn’t tuning but it seems like a good place… When streaming to the Adobe Flash player, it is important to try and avoid RTMPT (tunneling version of RTMP) as much as possible. RTMPT uses a polling mechanism which is very chatty and CPU intensive. We suggest employing a protocol rollover strategy so that only connections that need RTMPT use RTMPT. Is is described here:http://www.wowzamedia.com/forums/content.php?50
  14. On Windows 7, Server 2003 or 2008, it is probably best to turn off TCP auto tuning. There is information about how to do it here:How to disable Windows Vista TCP/IP auto-tuning
Bookmark the permalink.

Comments are closed.