There are two ways of limiting the process size:
In RHEL 4 and 5, edit the /etc/security/limits.conf file to add the following entry:
* soft as <Value>
For any other UNIX-based systems, including RHEL 4 and 5, specify the limits in the local/conf/runcmd_env.sh file, as follows:
ulimit -v <Value>
For example, you can specify a value of 6291456. Here, 6291456 is 6GB of address space = 6 * 1024 * 1024. The value here is in KBs. You can specify a value based on the maximum memory required for the VMware process.Note:
Limiting the process size causes the process to be terminated once it tries to exceed the set limit. As such, setting this limit may cause the domain managers exceeding the limit to be terminated by the operating system.