Xplat Part 2 – The Install

The Cross Platform Extensions were announced at the MMS and I wrote a post about the elements you need to test it. Here is the next part showing the install and the views and reports you get. I started this a month ago and it was delayed for various reasons so Cameron Fuller has a similar post out but as I did a lot of work I though I might as well finish it and publish it. Mine has a section on reporting.

xplat-evaluation-24june2008-pdf

This PDF is 3.3MB and 50 pages long. There are an awful lot of screen shots. This is designed not so much for people who want to install it (the existing documentation is very good) but people who do not want to or do not have the time to install it but would like to see what it looks like. This document goes through the install of the components and has screen shots of the various console areas and a section on reports out of the box and customising reports.

My summary:

I am very impressed by

1. How easy it was to install
2. How stable it appears to be 
3. How comprehensive it appears to be
4. How well it integrates with the console to make monitoring Linux and Unix the same as monitoring Windows so easy for existing operators to pick up

I believe that this is a breakthrough product for Operations Manager and Microsoft. I can see organisations that went for the framework and either ignored OpsMgr or relegated it to a role of being an element manager for AD or Exchange suddenly finding a lot to interest them with this product – especially when they see what it can monitor out of the box with no configuration.

I can not wait until the final release which is projected for early 2009.

Disk Space CPU % Report Table

One of the reports that MOM 2005 had out of the box was a summary report that showed a number of server parameters like CPU and disk space.

This does not exist in 2007 although it is constantly asked for by a number of organisations that I speak to and see comments about in the newsgroups. You can create performance graphs using the templates but they are graphs not a nice summary tables. I thought that it could not be done until I look at the reports from the Virtual Server MP. This has one report that pretty much sums up what a number of customers that I worked with have asked for.

What if you are not using Virtual Server? For this report it does not matter as this is the candidates for virtualisation report. The above report was done on test servers that are all running as guests on a VMWare ESX server without any trace of Microsoft Virtual Server.

The report comes with a number of parameters preset. All I did was up all the parameters to the maximum and that meant the report gave all the servers rather than just the candidates. All the performance counters are collected by the Windows 2000 and 2003 MP and not this one.

You can filter the number of servers using the parameters like CPU and memory or you can use group. The one caveat is that the date range is actual dates like in SRS so it does not appear you can use the 2007 controls to select dates like 1st day of month to last day which would make it better for scheduling. I saved it to favourites with All Computers and the parameters at max and so I just need to change the dates to run it again.

But it is a nice little report and worth importing the Virtual Server MP just so you can access this report. I have not tried it but I am sure in SRS you can copy it and make some amendments to customise it a bit more.

Why couldn’t the OpsMgr team have this as a standard report? I will leave the final comment to grumpy old man Victor Meldrew from the UK comedy “One Foot in the Grave“.

“I don’t believe it”.

Makes sense if you now the comedy program that it comes from.

Exchange 2007 SP1 MP

The updated management pack was release on 22nd June and is version 6.0.6278.12. So far so good. And as with all these things it is good to read the MP guide that is extracted as part of the MSI package before installing it.

Some key points (there are more in the guide)

The management pack was updated to support the renamed performance counters in Exchange Server 2007 Service Pack 1. Performance counters for the Database object were renamed to MSExchange Database.

Fixed some cluster issues.

The reports were updated to support non-US locales on the Reporting Server.

Nice to see that they have recognised that not everyone lives in the US. On the first point

With the release of Exchange Server 2007 Service Pack 1, some performance counters were renamed. Specifically, the Database object was renamed to MSExchange Database (this affects the Mailbox, Hub Transport and Edge Transport roles). The 6.0.6278.12 version of the Exchange Server 2007 Management Pack is tailored to only collect these performance counters on servers running Exchange Server 2007 Service Pack 1. When Operations Manager monitors Exchange Server 2007 RTM, agents on those servers will report missing performance counters. To resolve this problem, disable the rules looking for those counters on RTM servers, or upgrade all Exchange Servers to Service Pack 1.

If you are not running Exchange 2007 SP1 then be prepared for some work. I wanted to check a system running Exchange 2007 to see if it was SP1. The Help About only gives a build number and Add Remove programs does not mention SP1. The build number for SP1 is mentioned in the MP guide and the following web site gives the version number of all Exchange versions.  http://support.microsoft.com/kb/158530. Useful for people like me who go to various customers and need to work out what is what.

Before you import the MP there are a number of things to check. There are a couple of hotfixes if you are running on a cluster that need to be done – 951979 and 951380. But the one I wanted was 950583.

Install the agent update specified in Knowledge Base article 950853 on all Exchange-based servers managed by Operations Manager before importing the Exchange Server 2007 Management Pack. This update addresses an agent memory leak issue.

http://support.microsoft.com/kb/950853

But when you read this KB you can not download it. You need to request it from Microsoft support and the KB advises NOT to install it unless you really need to. The MP guide says to install it before doing the MP. I certainly would not want a memory leak on an Exchange server.

So read the MP guide BEFORE importing this MP.

It is nice to see the guide show a list of reports and what rule they depend on for collecting the information. This means that it is easy to know what to switch off if you never run that report to stop data being collected. All MP guides should have this information.

The one thing I like about the Exchange 2007 MP is that you no longer have to run the Exchange MP Wizard. It is now just a PowerShell script that you run. Much easier.

Memory, Memory, Memory

If there is one area that can help in an OpsMgr deployment it is memory on the RMS. And to do that usually means 64 bit OS. Windows 2003 Standard x64 supports up to 32 GB whereas x32 is only 4 GB.

Checkout the memory usage on this RMS.

Frightening really. If there is one area that I would like to see improved for SP2 it is in better performance for the OpsMgr components – especially the RMS and console. I would rather have less features that work really well rather than a host of features which don’t really work or are so slow they are unusable.

Delete Me

Well not literally of course!

When I am doing training to transfer across an OM installation I like to show some of the things it can do. It is better to use the onsite system that has been installed as the server names will be familiar. With MOM 2005 I used to use a virtual machine so I could demo how to do certain things. This was OK as when I finished I switched off the VPC without saving state so it rolled back to how I last saved it.

With 2007 it is heavier on the VPC (needed to buy a new laptop to run it properly!) but as they have changed how management packs work I find the following easier. I create an MP called Delete Me and then I demonstrate how OpsMgr can be used to create simple rules and monitors and such and make sure these all go into the Delete Me MP. When I am finished I can just delete the MP. Much quicker and simpler than firing up a VPC.

KB951979 (SP1 Hotfix)

Well after the fun and games with Microsoft Support I eventually received my hotfix. They state that you use it at your own risk, they password protect the download file, you then run it to get an MSI which you then run to get the 4 management packs. Why do they make it so hard?

If you want the full details on what it fixes Clive has it all as usual: http://blogs.technet.com/cliveeastwood/archive/2008/06/10/new-kb951979-sp1-hotfix-problems-occur-on-a-management-server-that-is-running-system-center-operations-manager-2007-service-pack-1-when-certain-management-packs-are-installed.aspx

Summary

1. The management server cannot collect the CSDVersion property from Windows Vista-based client computers.
2. The management server cannot collect operating system properties from Windows 2000 Service Pack 4-based client computers
3. The HandleCountThreshold agent monitor does not restart the HealthService service on the Microsoft Exchange Server agent even when the HandleCountThreshold value has been exceeded.
4. Audit Collection Services (ACS) events cannot be collected correctly.
5. Incorrect warnings are raised for Performance Data Source Module when some services in the system are disabled
6. Cluster discovery does not work correctly when the Virtual Server Name is a substring of the Physical Server Name

I am working at a client site with Windows 2000. I install the MPs and within a few minutes all the discovery information started getting populated. It has solved a CIMv2 issue I was having with a Windows 2003 server that kept erroring and did not collect the inventory. I was about to roll my sleeves up and get stuck into WMI but this cured it. Marvelous. But it has taken them over 14 months to get SCOM (a version 3 product) to do a basic inventory of some standard Windows stuff like server version and SP info. What ever happened to the Dynamic IT vision that Bob Muglia is still talking about (keynote at TechEd this week)? We need to see the product group becoming a lot more dynamic.

As for Performance Data Source Module this has been a bug since SP1 with the recommendation to switch it off with an override. It is nice to delete that override and not worry about it again. I hope.

The versions are 6.0.6278.19. Hopefully once they are happy with them they will just release them to the web site. I have not seen any issues since installing.

When importing you will get a warning.

This is because the internal library MP has rules that run on the management server. You get the warning sign when you import and if you click it it will show you the rules in question.

Nice touch to ensure that unscrupulous people don’t sneak rules into the MP.

SolarWinds free Exchange Monitor

Saw this at 4Sysops
http://4sysops.com/archives/solarwinds-exchange-monitor/

Nothing to do with SCOM but it does provide monitoring. Reminds me of Quest Spotlight. By the Quest do some free tools including Spotlight for Windows – http://www.quest.com/free-tools/.

SolarWinds Exchange Monitor is small and neat but it can only point to one server at a time and it does not remember the servers so you have to type in the name each time. You can configure the thresholds but it is only one set. So if you have a set for BE servers and another for FE servers then you can only have one set. But it does a nice “at a glance” view of services, mail queues, CPU utilisation and disk space. It only does Exchange 2000 and 2003. It is an 8.4 MB download.

This free desktop dashboard continuously monitors Microsoft® Exchange servers to deliver real-time insight into Exchange services, mail queue sizes, and host server health. With Exchange Monitor at your fingertips, you’ll be able to track Exchange health at-a-glance and ensure this mission-critical app never fails you.

SolarWinds free Exchange Monitor makes it easy to:

• Quickly identify and troubleshoot Exchange server problems, preventing email delays and calendaring issues
• Spot growing mail queues that can indicate bigger issues, like transport failures, Internet connection failures, and virus activity
• Leverage out-of-the-box settings based on best practices to start monitoring Exchange immediately
• Prevent performance issues before they impact users with built-in red, yellow, and green health indicators
• Monitor critical server health indicators, including disk space, CPU utilization, and memory utilization, to ensure optimal hardware functionality

SolarWinds Exchange Monitor

 

Quest Spotlight on Exchange

SCOM 2007 Exchange Computer view shows state of components.

In SCOM you can create your own performance views in dashboards. As many as you like. Here is a 4 pane dashboard for an Exchange server.

Hotfix KB951979

A few blogs have mentioned an upcoming hotfix including Clive Eastwood.

https://blogs.technet.com/cliveeastwood/archive/2008/06/10/new-kb951979-sp1-hotfix-problems-occur-on-a-management-server-that-is-running-system-center-operations-manager-2007-service-pack-1-when-certain-management-packs-are-installed.aspx

He says

Problems occur on a management server that is running System Center Operations Manager 2007 Service Pack 1 when certain management packs are installed. These management packs will not be posted on the Management pack catalog and are only available as a hotfix at this time.

As I am seeing a couple of these problems I followed the link and requested the hotfix. One of the fields to fill in was the product so I put in Operations Manager 2007 SP1.

Here is the reply I received from Microsoft Product Support.

The hotfix described in KB article 951979 is included in System Center Operations Manager 2007 Service Pack 1.

As you already have Service Pack 1, I recommend contacting Microsoft Support Professional.

As I am running SP1 and seeing these problems and the hotfix was just released on June 9th then it seems unlikely. And between some support person who does not have the decency to put their name in the e-mail and Clive Eastwood who is a PM and I have known him for years I am going to trust Clive on this one.

Definitely a bit more training needed for the mystery person in Microsoft Support.

Subscription Issues

The organisation I am currently working with has used Alert rules in MOM 2005 to create an event on the management servers when a new alert comes in. There is an agent from their framework package that picks up these events. This is an old trick from the MOM 2000 days but works remarkably well if you just want a one way connection. You have to remember to clear the alert when the incident is fixed otherwise if it happens again it will increment the repeat count and not fire the alert rule.

They want to carry on doing that with 2007 but there are no Alert rules. The obvious place to start is with subscriptions. As their old script created an event then the command notification is the pace to start. Using the command notification channel has been posted by Steve Rachui and using it with eventcreate has been posted by Stefan Stranger and

I used a number of parameters that went into the description field and the framework could pull those out and put them into the fields they wanted. However a few rules got the Alert Source parameter the wrong way around so the computer name and the object were swapped around. Therefore the Managed Entity (aka computer name) is wrong. Not great when you are automating these especially as the 2 main culprits are for agent heartbeat.

The Agent Source parameter is $Data/Context/DataItem/ManagedEntityPath$\$Data/Context/DataItem/ManagedEntityDisplayName$ which normally turns into something like w2k302.domain.co.uk\Microsoft(R) Windows(R) Server 2003 Standard x64 Edition

But for “Health Service Heartbeat Failure”  and “Failed to Connect to Computer” you get it the wrong way around Microsoft.SystemCenter.AgentWatchersGroup\w2k301.domain.co.uk

And in SQL 2005 Percentage Change in DB % Used Space part of the object is tagged to the ManagedEntityPath so you get w2k301.domain.co.uk;MSSQLSERVER returned.

This can be verified by creating a console view with Path and Source as the columns.

The eventcreate notification does the job except for monitors without alerts. Subscriptions are only triggered by alerts. There are some monitors that do not create alerts and worse they do not allow you to override that parameter. So if you are relying on notifications by e-mail, SMS or in this case events triggering the framework to create a ticket there will be some monitors that you will not get.

This is a problem that I discovered in the early betas and noted in my post about RC2 so it has been around a while. Some monitors do not create alerts but change the health of the state view. All rules create alerts but there is no way to have a column in the state view to show the effect they are having on the server so there is no one view that displays the total health of the estate. The only option that Microsoft could come up with is to use a Dashboard view with both views in it. Not great.

Stefan Stranger has confirmed this and the feedback from the Product Group is that this is “by design”. The trouble with that is that if you are not looking at the console you will not get the health state change as there is no alert. And as these are not documented that is not good. Even worse if you can not overrode them. In which case if you want that alert you need to disable the original monitor and recreate a new custom one (if you can).

Stefan sent these PS scripts to help which I amended as it is only unit monitors that are the issue. It does not matter that rollup or aggregate monitors do not alert.

get-monitor | where { ($_.AlertSettings -eq $null) -and ($_.xmltag -eq “UnitMonitor”) } | sort displayname

get-monitor | where { ($_.AlertSettings -eq $null) -and ($_.xmltag -eq “UnitMonitor”) } | export-csv c:\unitmonsno.csv

To get a list of monitors that do alert change the first -eq in the line to -ne.

get-monitor | where { ($_.AlertSettings -ne $null) -and ($_.xmltag -eq “UnitMonitor”) } | export-csv c:\unitmonsalerts.csv

When I run these on the management group there were 168 monitors without alerts (mainly from IIS 5 and 6 MPs) and 615 with alerts. That is why I switched  from display on screen to CSV.

You can also use MP Viewer to look at each MP. This has the ability to sort columns and you can see which ones are set not to alert. You can then make a decision on whether or not to do something about it.

I also have a problem with the way recipients and subscriptions are done but that is another post!

The Blog List

A while back I started a page at the side of my blog with a list of all the MOM and SCOM bloggers.

http://ianblythmanagement.wordpress.com/mom-blog-links/

The feeling was if you came across my blog by accident but wanted more on Operations Manager you could look at the list and add which ones you fancied.

As time has gone on it became unwieldy as more people started to blog, some have left and blog names have changed. I used to list them with names I would remember but it started to become difficult to compare my feeds with that list.

I have spent some time and revamped the lists so that the names are the same as the blog name. This makes it easier for me to check that the list is up to date with my RSS reader. I have also demoted blogs that have not been updated in a while to the bottom category – old blogs. But there are still some great posts in these. Perhaps someone should pull together all the good posts into a mega document so that they do not get lost.

The list is split into Microsoft and community bloggers plus I have a list for SCE and another for community bloggers who do good posts about SCOM but are not exclusive. This allows me to include Techlog amongst others now.

I named mine starting with my name first. In retrospect that was a good idea and wish that others did the same. Having a list in my reader that all start with System Center then gets cut off does not help! Be proud of your blog and put your name at the front so it is easy to find in a list.

I have also starred feeds that I personally think are useful to add. Sorry to anyone who is not on the list but it helps narrow it down for newcomers. I personally read every blog on that list and if you are keen on SCOM then you should too.