Greetings all:

I have a customer for whom I am doing an email migration for. We are migrating to Messaging Server 6.3 as part of the Sun Java Communications Suite 5. I understand that ZFS is supported in this version for the mail store but my customer has a number of questions that I would like to post here for comments. They are:

1) Filesystem - What filesystem will be used to store the data? Currently we are using a RAID 5 Direct Attached 3.0 TB array. What is the best way to set this up?
a) If ZFS is ready for prime-time we would prefer the use of it given UFS's limitations with 1.0 TB partition - 1 million files per 1.0 TB. If not what is best way to set up UFS given that we have about 1500 student accounts and 500 staff/faculty accounts?
b) Should data be on one partition with 2.5TB of space for data and .5TB space for logs? We also have a second 146GB disk on the v490 that could be used for logs if that is large enough and if i/o to the Direct-Attached array is an issue
c) tempfs appears useful for some files with JCS. Adding more to this in a bit.

2) Quotas - Related to question 1 for how filesystem is broken up if necessary. If using ZFS ignore section on benefits of single-copy email downside if broken into multiple partitions.
a) We would like the largest quota possible while first maintaining critical aspects - system performance, reliability and backups.
b) The average user with a 75MB quota is using approximately 44MB of disk space, for about 2050 users totaling 90GB of data.
c) Is is safe to assume that the same number of users using 500MB quota (assuming 100% usage) will be using 1.026TB of space?
d) Assuming c) is correct, is it safe to assume that giving a 1GB quota will be using 2.052TB?
e) Is 80% of quota a fair estimate of a typical user? Any stats (Gartner, other implementations) to back up a typical use case?
f) If e) is correct, is it safe to give quotas in the typical form
2007-2008 - 1GB
2008-2009 - 1.25 GB
2009-2010 - 1.50 GB
Note: Every Year approximately 400 accounts are deleted on December 31st.
g) Are there any performance issues with assumptions a)-f)?
h) Are there any backup issues with assumptions a_-f)?
i) Are there any other considerations to consider?
j) How hard is it to move a batch of users (say 100) to a new partition on a NAS, SAN, or additional direct-attached users when the filesystem gets full? How long would it take to move 100 1GB accounts? Would it affect system performance in any way? Is this moot with ZFS?
k) Since single-copy efficiency is only used per partition (as stated in Sun's docs) ,is there a way to determine current single-copy efficiency statistics that can be used if multiple less than 1 Terabyte UFS partitions are necessary?
l) If using about 4 million files with 90GB accounts, it it safe to assume 100 million files for 2TB of data? If using ZFS, is cloning or similar ZFS/JCS backup method viable? If using UFS, is imsbackup, volcopy, or other Solaris imaging viable? Any pitfalls to this approach?

 

 

Hi,


You understand incorrectly. ZFS has not been officially certified for use with messaging server 6.3 - work is being carried out to 'certify' ZFS and SAM-FS for the next release. Other customers are using ZFS and haven't reported significant issues but that's not to say they don't exist (issues that is).

The advice I can offer is if you plan on running ZFS, make sure your system is patched to at least Solaris 10u3 which includes a number of ZFS performance improvements.

Given that messaging server relies so highly on the behavior and speed of an underlying file-system we need to run any number of tests to make sure that messaging server not only works but doesn't hit significant bottlenecks.

They are:

That is up to the customer as to what filesystem they run. We support UFS and VxFS on Solaris.

Make sure that you systems have fast write times - this is essential. Also you wouldn't want to have a single 3TB partition (file-system) as any filesystem corruption would cause the entire store to be unavailable.


Create multiple partitions of a few hundred GB each until ZFS is 'ready-for-prime-time'.


If you can put your logs and mailbox database directory on fast direct-attached that can improve performance. But it would depend on what the load of the system is. One v490 to cater for that number of students/staff shouldn't be a problem. I personally ran a single v480 for ~10,000 staff accounts and another for ~40,000 student accounts without them breaking a sweat.


Set store.dbtmpdir to point to /tmp/mail-store/

That will store the database temporary files on a tmpfs filesystem which improves performance.


Ok, but the most critical aspect is growth and how you intend to deal with growth. The biggest issue with direct-attach is running out of spare slots etc. and expanding file-systems to cater for store growth.

Remember also that filesystems start to perform badly (fragmentation issues) as they reach high utilisation (95%+) so you don't want to get near this value.


This is very much site dependant. In my case for a University staff population we were massively oversubscribed (amount of possible usage vs. actual usage) simply because a number of accounts got very little use.

One thing I would implement is some kind of default spam filtering so that 'spam' emails are put into a 'Spam' folder which is cleaned up every 30 days etc. That stopped growth tremendously in our student accounts which we had over 80,000 and for which a large number of students never checked (or used as spam-catches :( )


Already addressed above.


My personal experience at a University is that usage was much less then that. We tended to work on the add-more-space-as-required philosophy. So perhaps start with less space but have documented procedures in place on how to increase should the need eventuate.


Yes. Larger quotas result in larger inboxes which results in larger imap processes which requires more imap processes (until 64bit version is released) and potentially more I/O for backup/imexpire operations etc.


They will be larger and take longer? I'm not sure what else you mean? Also consider how you are going to restore in the case of a DR scenario, I know with legato we found it would take 2 weeks to restore the systems from scratch on a per-file level.

Consider also taking block-level backups if your backup software can handle it.


You can use the mboxutil command to shift between messaging partitions which can be mounted to different file-systems/san partitions. Make sure you use the 'relinker' command to restore the hard-links broken during the process.

This procedure should of course be tested BEFORE using on real accounts.


Do it and find out - thats a 'how long is a piece of string' question.


refer to relinker utility documentation. single-copy efficiency also works on the behaviour of your organisation (do you send out bulk-emails to 1 recipient/email or many recipients/email).


No idea.

Hope this helps.

Shane.

 

 

Shane,

Thank you for the in-depth response to all my customer's questions. I appreciate it.

My understanding of ZFS being supported in 6.3 came from jay_plesset's response in this post: http://forum.java.sun.com/thread.jspa?threadID=5175709

Jay, if you are out there, care to comment?

 

 

I was confused and incorrect. Shane is correct. Official support is due next release. However, some customers are indeed using ZFS, and haven't reported any issues big enough to inspire any warnings to us support folks.

 

 

Hi,

As an aside, I actually did some back-to-back testing to see just what impact ZFS had on overall numbers of IOps and read/write blocks compared to UFS given the fundamental differences in the file-systems.

The results were a bit of a mixed bag, wth ZFS having less IOps and for smaller files (small emails) and less blocks written - so in general larger read/write operations and more efficiently handled - which is good. For larger files (> 100K say) UFS had less overall data written but still had small read/write operations - so ZFS had more 'overhead'.

Backups were affected as a result with ZFS having an additional 10% overhead in the amount of data read off disk compared to UFS but nearly 50% less read operations, so it may be performance is better (disks prefer less larger reads).

How this works out in real life is going to depend on a lot of factors - so I suggest you benchmark against a ZFS and a UFS partition with the same simulated load.

One thing you definitely want to do though is to use LMTP from your MTA front-end to your store backend, this will save considerably on the amount of data written to disk and also the number of LDAP lookups.

Regards,

Shane.

 

 

When you say that ZFS will be supported in the "next release", do you mean the next core patch to 6.3, or the release of 6.4 (which will be approximately when?)

We're interested in investigating the possibility of using ZFS file system compression to reduce disk space. We saw awesome results with using ZFS fs compression in our log repository (60% space reduction, no performance loss.) We're estimating that we could get a 25-35% space reduction on the email stores, but we don't know what effect this would have on performance and reliability. Do you have input on this idea?

How does using LMTP reduce disk usage? Do you mean that it eliminates the need for MTA queues on the stores? But that is transient data and doesn't add up to much. Does LMTP actually reduce disk usage on the store message repository? Does it have something to do with message linking?

 

 

Just wanted to add a little something about ZFS. I am not using on my email server, but in a limited way on another system. Make sure you have lots of memory. ZFS will use it all. It is supposed to release it, and it does, but not fast enough for some applications. I am occasionally getting out of memory on some of my applications the first time they are run and then they will run the second time.

 

 

Hi,


Support is slated for the next full release (namely 6.4 or it could be 7.0.. who knows) which is not going to be before the start of next year.


I would say that compression is going to naturally have an impact on CPU usage (to compress/decompress) and also may impact the responsiveness of file-system which can reduce the throughput of messaging server.

What I would suggest is that you consider separating off the smaller folder index files from your message files - and just compress the message file partitions. The store. index files require fast-as-possible access.

You can use the following parameters to dictate the location of the store. files from the .msg files:

store.partition..messagepath
store.partition.*.path

At the end of the day if you want to go down this path.. please load-test and compare.


Not overall usage but data written to disk (I/O). With SMTP you need to queue the email first to stable storage and then eventually relocate that queued email across to the users partition.

LMTP removes the requirement for the relocation since the email is delivered directly to the users partition and hence reduces I/O since you don't need to double-write the same data.

Regards,

Shane.

 

 

This is a good discussion since I am looking at the same issue (on a smaller scale).

What sort of ZFS related mail store issues have you been seeing?

Saying 'any issues big enough' makes me confident on one hand but nervous on the other hand as that implies that there are some issues, just not major issues.

Would this issue (ZFS and Msg Server) be part of why Cluster 3.2 support is not officially there yet for JCS 5 (and is this also coming in the next release of Msg Server rather than a point/core release/patch?

 

 

I've personally heard of none, nor have I seen any myself.


Basically, if there are issues, we don't know of them. What it really means is that if you run across any issues, you will be working in "unsupported" terratory, and our Support folks are likely to tell you that. . . We cannot accept bugs filed against 6.3 for any problems found with ZFS.

Official support means we need to do full QA with it. JCS5 didn't get that with Cluster 3.2, so it's not supported. Ditto ZFS.

 

 

Hi,


Neither have I - although I know of general ZFS performance issues which are being addressed with each release of Solaris 10.

It breaks down to.. it's new and untried (for very large messaging sites at least) so we (Sun) need to step carefully. The following document has some interesting information on ZFS though:

http://www.sun.com/emrkt/campaign_docs/expertexchange/knowledge/solaris_zfs_perf.html#10


... doesn't mean we won't log a bug and look at fixing it, but if to resolve the bug means rewriting fundamental components of an existing release then Support folks will push back.


Ditto IE7 - which wasn't released until way after the beta program had completed etc.

Regards,

Shane.

 

 

I understand and appreciate both (yourself and Jay's) answers on this.

Does something like Cluster 3.2 certification generally happen with a major release (i.e. JCS 6 or whatever the next one will be called, 5.1, etc) or as a point release type of deal?

I would imagine that the answer would be 'it depends' where 'depends' is how different product revision X is from Y, etc and how many installations it would impact.

 

 

Hi,


Major releases usually. Although this all depends on the kind of pressures that customers apply; the more customers ask for it the quicker it happens.


Usually depends on how much time the various groups/people have to test and fix any bugs that come up - time being a function of priority :)

Regards,

Shane.

 

 

Hi,

Sun Cluster 3.2 and ZFS are now supported with Messaging Server 6.3.

Further details available at:

http://blogs.sun.com/SC/entry/java_messaging_server_6_3

and

http://msg.wikidoc.info/index.php/Messaging_Server_ZFS_Support

Regards,

Shane.