To R2 or not to R2?

That is indeed the question.

This was something that I was concerned about but with the release of the new Exchange 2007 MP it is confirmed – this is an R2 only MP.

This is great if you are on R2 but as R2 is a Software Assurance release there will be organisations that bought OpsMgr without SA and so will not be able to use this MP. This brings up the question of support. After all the Microsoft standard for support is 5 + 5 which means that MOM 2005 is still supported but how many new 2005 MPs have you seen?

If the OS or application is only using some events and perfmon counters it can not be too hard to create MPs for MOM 2005, OpsMgr 2007 SP1 and OpsMgr 2007 R2. Notice that I don’t expect anyone to create an MP specifically for OpsMgr 2007 RTM. That is because SP1 is free and fixes so much with the RTM version that you would be foolish not to implement it. But creating 3 MPs would be hard work for complex products that use health models and lots of synthetic transactions. Like the Exchange MP for example. Still I think it is something that MP creators should aspire to. It would be interesting if there were any figures on deployments of MOM 2005 v 2007 and how how many organisations bought 2007 with SA. If Microsoft can show that 90% of organisations have moved to 2007 and most of them have SA then there is a stronger case for R2 only MPs. But there is still a need to support people on older versions. This presents an interesting challenge for Microsoft. I still come across organisations using older OSes and applications. As much as Microsoft would like everyone to move to the latest version the fact is that they don’t but still need to be supported as long as they are in the 5 + 5 timeframe.

It would be nice to see a white paper covering creating MPs for the 3 versions to help MP authors. I think that with the modular nature of OpsMgr it should be possible to create the base MPs for SP1 and have addition MP files that you can import if you have R2 that utilises the extra features and functions in R2. That would mean a different approach to creating a Management Pack but it would help in the long run. Mind you testing would be more complicated. And Microsoft’s track record on testing MPs before releasing them has not been that spectacular.

By the way I have been informed that there will be a native Exchange 2007 MP for SP1 that is due out in Q3. Good news on that front.

Exchange 2007 MP Oddities

This post relates to OpsMgr 2007 SP1 with Exchange 2007 MP v6.0.6461.0.

Normally I ensure that there is an agent on each Exchange server before deploying the MP. This ensures that the underlying OS and IIS are OK before getting involved in the Exchange MP. But on this occasion the customer had one Exchange server that they did not want to have an agent on as it was a temporary server used for migration and was being decommissioned in a month.

After deploying the MP I received the usual alerts from the Test-OwaConnectivity, Test-ActiveSyncConnectivity, and Test-WebServicesConnectivity rules which indicated that the script New-TestCasConnectivityUser.ps1 needs to be run to create the test mailboxes as per the MP Guide. This was done but I was still seeing some alerts but looking closely at the detail they showed that although the alerts were coming from the Exchange server with the agent they were actually referring to the mailbox server that did not have the agent. The Cas_GUID part showed that the GUID was one that I had not seen on running the script originally. I ran the New-TestCasConnectivityUser.ps1 script on the server without an agent and that resolved those alerts. It created the CAS_GUID account with the same GUID that I had seen in the alert.

Initially I could not run the script as the server had a recovery storage group on it. As that can not have mailboxes added the script fails. You need to remove the recovery storage group before running the script on that server,

http://technet.microsoft.com/en-us/library/bb125197.aspx

You need to run the commands get-storagegroup and get-mailboxdatabase to get the information to get the right one to remove.

Another alert I saw was that there was an error for the external OWA test. In the MP Guide it says

To set an external URL, you must run the Set-OwaVirtualDirectory Exchange Management Shell command. The syntax of the command is: set-owavirtualdirectory “<Server name>\owa (Default Web site)” -externalurl:https://<Fully Qualified Domain Name>/owa

I checked and found that this was already set correctly. I then discovered that the Exchange server needed Rollup 7 whereas these ones only had Rollup 6. This is because the domain name that the client uses is different from the FQDN that they have and so have a disjoint namespace.

Note that Rollup 8 has been released.

Discoveries

I was investigating a problem with a SQL server having 100% CPU spikes. I found that Sysinternals Process Monitor invaluable in helping me see what was going on. What was happening was that a large number of scripts were running at the same time and as it was a SQL cluster with multiple instances (this customer was doing a SQL Server consolidation) then it was exasperated as the scripts run for each instance at the same time. This has been confirmed by Microsoft as being a problem with the SQL MP (v6.0.6460.0) when running on a cluster with multiple instances all running off one active node.

To improve the situation I created a group with the cluster members and went through all the discoveries that had generic targets (like  Windows Computer or Windows Server) and put an override on them to stop them. This cluster is only going to be used for SQL Server and so there is no need to discover anything else. The main ones that really helped were eliminating the discoveries of SRS, Integration and Analysis Services as those were getting run for each instance. The active node is down to running about 15% which the SQL DBA thinks is dreadful but at least there are no 100% spikes now.

In my investigationI found some strange things. The IBM Director Agent script was set to run every 20 seconds – targeted to Windows Computer. I don’t even run discoveries that fast on a demo system! That was definitely changed.

Here is a list of more reasonable times for discoveries and what they are in seconds that you can use for overrides.

1 hour        3600 seconds
2 hours        7200 seconds
4 hours        14400 seconds
8 hours        28800 seconds
12 hours    43200 seconds
24 hours    86400 seconds
1 week        604800 seconds

I have always thought that one of the advantages of the application since MOM 2000 has been its automatic discovery and downloading of MP rules. This meant that if someone installed IIS on a server and forgot to tell the monitoring team it would not matter as the discovery would ensure that IIS was found and the rules downloaded. With 2007 using Cscript it seems that there is more chance of hitting 100% CPU which was pretty much unheard of in 2000 and 2005. Yet I have seen threads on forums about this for the AD and DNS MPs and now SQL on clusters with multiple instances.

In contrast I have been looking at the beta of the new Exchange 2007 MP which has been written for 2007 rather than converted from 2005. One of the things that struck me immediately when reading the MP Guide was that the majority of discoveries are switched off by default and when they are enabled the default is a rather more sensible 24 hours. After all how often do you change server roles in a production environment? This is a philosophical change as in the past all discoveries were targeted to all servers. The only discovery in this MP that does this is a light weight discovery that just checks registry keys. Once that has been done then the other discoveries (when switched on) are targeted at just those servers. That type of behaviour is seen in a number of MPs where a general discovery is run against Windows Computer but then other specific discoveries are targeted towards the class that is discovered. Obviously this puts less load on servers that are not running that application – especially if the discovery uses big scripts and/or WMI.

I like this idea. I have suggested in the past that the Exchange MP is split into two with the basics (event monitoring) in one and the advanced stuff that needs configuration (synthetic transactions) in a second. While this is not how the Exchange MP has been done it is split into multiple roles so you can just install the mailbox monitoring or the CAS or Hub MPs. This will make it easier to tune as you can put in one bit at a time.

I would recommend that you have a look at your discoveries and how often they are running and ask yourself what frequency is good for your environment. I would suggest a long period like a day or even a week for most as you can always create an override for a shorter period if you need to speed things up temporarily.

I was hoping to add the almost obligatory 1 line PowerShell script to show you how to get that information but although there is a get-discovery command it does not include the actual frequency of the discovery. You can always use the excellent MP Viewer (from Boris Yanushpolsky) but that can only look at 1 MP at a time. But it can examine an MP before you import it. It has a node for discoveries and will tell you the target, whether it is enabled or not and the all import frequency (in seconds).

Help on Exchange Reports in the Guide

One of the annoying things with doing reports in OpsMgr 2007 is getting the target right so you can get data in the report. I posted about this and Kevin Holman has also done a post. I am pleased to see that in the latest MP Guide for Exchange 2003 (released a few days ago), not only do they tell you what reports will be installed but they have a table showing what the targets are and what you should search for to get those targets. See How To Run Reports near the end of this section of the online version of the MP Guide – http://technet.microsoft.com/en-us/library/cc511014.aspx.

This is a great start and will be of immense use for people who have been frustrated trying to get these reports to show data. Now we just need all the other MP Guides to add similar information.

Exchange 2007 SP1 MP

The updated management pack was release on 22nd June and is version 6.0.6278.12. So far so good. And as with all these things it is good to read the MP guide that is extracted as part of the MSI package before installing it.

Some key points (there are more in the guide)

The management pack was updated to support the renamed performance counters in Exchange Server 2007 Service Pack 1. Performance counters for the Database object were renamed to MSExchange Database.

Fixed some cluster issues.

The reports were updated to support non-US locales on the Reporting Server.

Nice to see that they have recognised that not everyone lives in the US. On the first point

With the release of Exchange Server 2007 Service Pack 1, some performance counters were renamed. Specifically, the Database object was renamed to MSExchange Database (this affects the Mailbox, Hub Transport and Edge Transport roles). The 6.0.6278.12 version of the Exchange Server 2007 Management Pack is tailored to only collect these performance counters on servers running Exchange Server 2007 Service Pack 1. When Operations Manager monitors Exchange Server 2007 RTM, agents on those servers will report missing performance counters. To resolve this problem, disable the rules looking for those counters on RTM servers, or upgrade all Exchange Servers to Service Pack 1.

If you are not running Exchange 2007 SP1 then be prepared for some work. I wanted to check a system running Exchange 2007 to see if it was SP1. The Help About only gives a build number and Add Remove programs does not mention SP1. The build number for SP1 is mentioned in the MP guide and the following web site gives the version number of all Exchange versions.  http://support.microsoft.com/kb/158530. Useful for people like me who go to various customers and need to work out what is what.

Before you import the MP there are a number of things to check. There are a couple of hotfixes if you are running on a cluster that need to be done – 951979 and 951380. But the one I wanted was 950583.

Install the agent update specified in Knowledge Base article 950853 on all Exchange-based servers managed by Operations Manager before importing the Exchange Server 2007 Management Pack. This update addresses an agent memory leak issue.

http://support.microsoft.com/kb/950853

But when you read this KB you can not download it. You need to request it from Microsoft support and the KB advises NOT to install it unless you really need to. The MP guide says to install it before doing the MP. I certainly would not want a memory leak on an Exchange server.

So read the MP guide BEFORE importing this MP.

It is nice to see the guide show a list of reports and what rule they depend on for collecting the information. This means that it is easy to know what to switch off if you never run that report to stop data being collected. All MP guides should have this information.

The one thing I like about the Exchange 2007 MP is that you no longer have to run the Exchange MP Wizard. It is now just a PowerShell script that you run. Much easier.

SolarWinds free Exchange Monitor

Saw this at 4Sysops
http://4sysops.com/archives/solarwinds-exchange-monitor/

Nothing to do with SCOM but it does provide monitoring. Reminds me of Quest Spotlight. By the Quest do some free tools including Spotlight for Windows – http://www.quest.com/free-tools/.

SolarWinds Exchange Monitor is small and neat but it can only point to one server at a time and it does not remember the servers so you have to type in the name each time. You can configure the thresholds but it is only one set. So if you have a set for BE servers and another for FE servers then you can only have one set. But it does a nice “at a glance” view of services, mail queues, CPU utilisation and disk space. It only does Exchange 2000 and 2003. It is an 8.4 MB download.

This free desktop dashboard continuously monitors Microsoft® Exchange servers to deliver real-time insight into Exchange services, mail queue sizes, and host server health. With Exchange Monitor at your fingertips, you’ll be able to track Exchange health at-a-glance and ensure this mission-critical app never fails you.

SolarWinds free Exchange Monitor makes it easy to:

• Quickly identify and troubleshoot Exchange server problems, preventing email delays and calendaring issues
• Spot growing mail queues that can indicate bigger issues, like transport failures, Internet connection failures, and virus activity
• Leverage out-of-the-box settings based on best practices to start monitoring Exchange immediately
• Prevent performance issues before they impact users with built-in red, yellow, and green health indicators
• Monitor critical server health indicators, including disk space, CPU utilization, and memory utilization, to ensure optimal hardware functionality

SolarWinds Exchange Monitor

 

Quest Spotlight on Exchange

SCOM 2007 Exchange Computer view shows state of components.

In SCOM you can create your own performance views in dashboards. As many as you like. Here is a 4 pane dashboard for an Exchange server.

Exchange 2007 SP1 MP in the Works

I saw this in the newsgroup and thought it was worth passing on to a wider audience.

From Åke Pettersson [MSFT] 21st May 2008.

Hello, so let me try to give you a bit of context around what we are trying to do, and the reasons behind the schedule changes.

With the release of the updated Management Pack we want to address as many critical issues as possible that affect Exchange monitoring with OpsMgr. Some of these are fixed by updating the MP and some are fixed by updating OpsMgr components (post-SP1 hotfixes). We decided to ship the MP only when those updated OpsMgr components are also available, in order to give customers the best possible solution. We have also made sure to document those hotfixes in the MP Guide, hopefully making it easier for customers to address the critical issues at once.

We are working hard to deliver the hotfixes. I understand the frustration with the slipping schedule, part of it is due to the testing that the hotfixes go through before release, to ensure quality.

I hope that at least explains a bit what we are trying to do and where we are at.
thanks
Åke