Implementing shared memory resource controls on Solaris hosts

With the availability of the Solaris 10 operating system, the way IPC facilities (e.g., shared memory, message queues, etc.) are managed changed. In previous releases of the Solaris operating system, editing /etc/system was the recommended way to increase the values of a given IPC tunable. With the release of Solaris 10, IPC tunables are now managed through the Solaris resource manager. The resource manager makes each tunable available through one or more resource controls, which provide an upper bound on the size of a given resource.

Merging the management of the IPC facilities into the resource manager has numerous benefits. The biggest benefit is the ability to increase and decrease the size of a resource control on the fly (prior to Solaris 10, you had to reboot the system if you made changes to the IPC tunables in /etc/system). The second major benefit is that the default values were increased to sane defaults, so you no longer need to fiddle with adjusting the number of message queues and semaphores when you configure a new Oracle database. That said, there are times when the defaults need to be increased. If your running Oracle, this is especially true, since the default size of the shared memory resource control (project.max-shm-memory) is sized a bit low (the default value is 254M).

There are two ways to increase the default value of a resource control. If you want to increase the default for the entire system, you can add a resource control with the desired value to the system project (projects are used to group resource controls). If you want to increase the value of a resource control for a specific user or group, you can create a new project and then assign one or more resource controls to that project.

So lets say that you just installed Oracle, and got the lovely “out of memory” error when attempting to create a new database instance. To fix this issue, you need to increase the amount of shared memory that is available to the oracle user. Since you don’t necessarily want all users to be able to allocate gobs of shared memory, you can create a new project, assign the oracle user to that project and then add the desired amount of shared memory to the shared memory resource control in that project

To create a new project, the projadd utility can be executed with the name of the project to create:

$ projadd user.oracle

Once a project is created, the projmod utility can be used to add a resource control to the project (you can also edit the projects file directly if you want). The following example shows how to add a shared memory resource control with an upper bounds of 1GB to the user.oracle project we created above:

$ projmod -sK “project.max-shm-memory=(privileged,1073741824,deny)” user.oracle

To verify that a resource control was added to the system, the grep utility can be run with the project name and the name of the system project file (project information is stored in /etc/project):

$ grep user.oracle /etc/project
user.oracle:100::::project.max-shm-memory=(privileged,1073741824,deny)

You can also check the value of a resource control with the prctl utility. Prctl takes a pid as an argument and optionally the name of the resource control to print (prctl will print the value of all resource controls by default):

$ prctl -n project.max-shm-memory $$

process: 585: -sh
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
project.max-shm-memory
        privileged      1.00GB      -   deny                                 -
        system          16.0EB    max   deny                                 -

The resource control facility is amazingly cool, and the folks who manage the Princeton Solaris support repository did a great job documenting it.

Capping a Solaris processes memory

Solaris 10 introduced numerous capabilities, including the ability to use memory caps to limit the amount of memory available to a project. Memory caps are configured through the project(4) facility, and use the rcap.max-rss resource control to limit the amount of memory that a project can consume. Memory caps are enforced by the rcapd daemon, which is a userland process that periodically checks process memory usage, and takes action when a process has exceeded it’s alloted amount of memory. To use memory caps on a server or inside a zone, the rcapadm utility needs to be run with the “-E” (enable memory caps) option to enable memory caps:

$ rcapadm -E

In addition to starting rcapd, the capadm utility will enable the SMF services to start rcapd when the system boots. To see if memory caps are enabled on a system, the rcapadm utility can be run without any arguments:

$ rcapadm

                                 state: enabled
      memory cap enforcement threshold: 0%
               process scan rate (sec): 15
            reconfiguration rate (sec): 60
                     report rate (sec): 5
               RSS sampling rate (sec): 5

After memory capping is enabled, the projmod utility can be used to configure memory caps. To configure a 512MB memory cap for all processes that run as the user apache, the projmod utility can be run with the “-K” option, and the rcap.max-rss resource control set to the amount of memory you would like to assign to the project:

$ projmod -s -K rcap.max-rss=512MB user.apache

This will add a new entry similar to the following to the project database, which is stored in the file /etc/project:

$ grep user.apache /etc/project
user.apache:100:Apache:apache::rcap.max-rss=536870912

Once a project is configured, you can enforce a memory cap in two ways (there may be more, but these are the two methods I have come across while reading the RM documentation). The first method uses the newtask utility to start a process in a project that has been configured with memory caps. The following example shows how to start the apache web server in the user.apache project, which was configured above:

$ /usr/bin/newtask -p user.apache /home/apps/apache/httpd/bin/httpd -k start

The second way to enforce a memory cap is to force a user to establish a new login session. If the user has been added to the project database, they will inherit the resource controls that are associated with their user id in /etc/project. To view the project a user is assigned to, the id command can be run with the “-p” option:

$ su – apache
Sun Microsystems Inc. SunOS 5.10 Generic January 2005

$ id -p
uid=103(apache) gid=1(other) projid=100(user.apache)

Once a process is started and associated with a project that has memory caps configured, you can use the rcapstat utility to monitor memory usage, and the paging activity that occurs due to the processes in the project utilizing more memory than has been alloted to them:

$ rcapstat 10

    id project         nproc    vm   rss   cap    at avgat    pg avgpg
   100 user.apache        15  266M  164M  512M    0K    0K    0K    0K
   101 user.mysql          1   59M   11M  256M    0K    0K    0K    0K
    id project         nproc    vm   rss   cap    at avgat    pg avgpg
   100 user.apache        15  266M  164M  512M    0K    0K    0K    0K
   101 user.mysql          1   59M   11M  256M    0K    0K    0K    0K

Memory caps are super useful, but they do have a few issues. The biggest issue is that shared memory is not accounted for properly, so processes that use shared memory can suck up more memory that the amount configured in the memory cap. The second issue is that you can’t use memory caps in the global zone to limit how much memory is used in a local zone. Both of these issues are being worked on by Sun, and hopefully a fix will be out in the coming months.