To R2 or not to R2?

That is indeed the question.

This was something that I was concerned about but with the release of the new Exchange 2007 MP it is confirmed – this is an R2 only MP.

This is great if you are on R2 but as R2 is a Software Assurance release there will be organisations that bought OpsMgr without SA and so will not be able to use this MP. This brings up the question of support. After all the Microsoft standard for support is 5 + 5 which means that MOM 2005 is still supported but how many new 2005 MPs have you seen?

If the OS or application is only using some events and perfmon counters it can not be too hard to create MPs for MOM 2005, OpsMgr 2007 SP1 and OpsMgr 2007 R2. Notice that I don’t expect anyone to create an MP specifically for OpsMgr 2007 RTM. That is because SP1 is free and fixes so much with the RTM version that you would be foolish not to implement it. But creating 3 MPs would be hard work for complex products that use health models and lots of synthetic transactions. Like the Exchange MP for example. Still I think it is something that MP creators should aspire to. It would be interesting if there were any figures on deployments of MOM 2005 v 2007 and how how many organisations bought 2007 with SA. If Microsoft can show that 90% of organisations have moved to 2007 and most of them have SA then there is a stronger case for R2 only MPs. But there is still a need to support people on older versions. This presents an interesting challenge for Microsoft. I still come across organisations using older OSes and applications. As much as Microsoft would like everyone to move to the latest version the fact is that they don’t but still need to be supported as long as they are in the 5 + 5 timeframe.

It would be nice to see a white paper covering creating MPs for the 3 versions to help MP authors. I think that with the modular nature of OpsMgr it should be possible to create the base MPs for SP1 and have addition MP files that you can import if you have R2 that utilises the extra features and functions in R2. That would mean a different approach to creating a Management Pack but it would help in the long run. Mind you testing would be more complicated. And Microsoft’s track record on testing MPs before releasing them has not been that spectacular.

By the way I have been informed that there will be a native Exchange 2007 MP for SP1 that is due out in Q3. Good news on that front.

Create Events

Something that is handy for demos and testing systems is to create events in order to check that rules and monitors work and create alerts to order. Usually if you want an event to happen in the event log it never will. As soon as you turn your back thousands happen.

I was confronted with this recently as a customer wanted to do a functional test after installation and want to create events from some of the key MPs to ensure that they worked. Using MP Viewer I can look into the MPs, look at the rules and find out which event numbers with source and event log are in an MP and enabled. Normally for demos I use eventcreate.exe as it is part of the OS. Unfortunately this only does events up to 1000. So I looked around to see what I could find. It was difficult to search for as most searches tend to respond with tools to monitor and manipulate event logs. So there may be other tools out there.

1 Eventcreate.exe

Pros – included in Windows 2003, 2008, XP and Vista; can do remote to another server; can be used in a batch file.

Cons – only up to ID 1000, only Application and System logs

2 Logevent.exe

Pros – can be included in a batch file; can do all event IDs; can do remote.

Cons – Can only do Application log; was part of Windows 2000 Resource Kit but no longer shown in that download area.

3 VBScript

Pros – run from anything that can do VBscript

Cons – Incredibly limited; only one source – WSH; only Application log; event ID equals type of event (0 for info, 1 for warning etc).

4 PowerShell

Pros – Clever use of interactive PowerShell script from Stefan Strange with additions from Ken

Cons – interactive (would need extra work to take parameters to be used in a batch file); can only do local server (but could be extended with more code) and so needs PowerShell installed

5 Event Create (MOM 2005 Resource Kit)
http://technet.microsoft.com/en-us/opsmgr/bb498240.aspx
http://download.microsoft.com/download/d/4/b/d4bfc32c-d8d2-4541-8356-4c6359eecbb4/MOM2005ResKit.msi

Pros – GUI front end that can do remote; any event logs; all IDs

Cons – can not be used in batch file; msi needs .Net v1 to extract its files

6 OpsMgr Script

Pros – simple to create a timed script to run as at a regular interval; can use OpsMgr targeting; could be created as a VB script to run as batch file

Cons – can only log to Operations Manager log; can only use Health Service Script as a source; needs OpsMgr agent in order to work

Example showing how to ping.

Summary

If you need to create events and are happy with the limitations then eventcreate.exe is easy and is included in all the latest OSes. For a demo the Event Create GUI from the MOM 2005 Resource Kit is great. If you can get hold of it logevent can automate event creation (as long as it is the Application event log). Highly recommended is the PowerShell script from Stefan and Ken. With some more work it would be able to do everything.

Updated 26/8/08 – Changed references to logevent.exe as it is not included in MOM Resource Kit as I originally thought. It was in my 2005 VM but when I did a new install only the GUI version is there.

It would be nice to have a single tool like eventcreate.exe but without the limitation on event logs and event IDs. Anyone want to create a tool like that?

Disk Space CPU % Report Table

One of the reports that MOM 2005 had out of the box was a summary report that showed a number of server parameters like CPU and disk space.

This does not exist in 2007 although it is constantly asked for by a number of organisations that I speak to and see comments about in the newsgroups. You can create performance graphs using the templates but they are graphs not a nice summary tables. I thought that it could not be done until I look at the reports from the Virtual Server MP. This has one report that pretty much sums up what a number of customers that I worked with have asked for.

What if you are not using Virtual Server? For this report it does not matter as this is the candidates for virtualisation report. The above report was done on test servers that are all running as guests on a VMWare ESX server without any trace of Microsoft Virtual Server.

The report comes with a number of parameters preset. All I did was up all the parameters to the maximum and that meant the report gave all the servers rather than just the candidates. All the performance counters are collected by the Windows 2000 and 2003 MP and not this one.

You can filter the number of servers using the parameters like CPU and memory or you can use group. The one caveat is that the date range is actual dates like in SRS so it does not appear you can use the 2007 controls to select dates like 1st day of month to last day which would make it better for scheduling. I saved it to favourites with All Computers and the parameters at max and so I just need to change the dates to run it again.

But it is a nice little report and worth importing the Virtual Server MP just so you can access this report. I have not tried it but I am sure in SRS you can copy it and make some amendments to customise it a bit more.

Why couldn’t the OpsMgr team have this as a standard report? I will leave the final comment to grumpy old man Victor Meldrew from the UK comedy “One Foot in the Grave“.

“I don’t believe it”.

Makes sense if you now the comedy program that it comes from.

The Blog List

A while back I started a page at the side of my blog with a list of all the MOM and SCOM bloggers.

http://ianblythmanagement.wordpress.com/mom-blog-links/

The feeling was if you came across my blog by accident but wanted more on Operations Manager you could look at the list and add which ones you fancied.

As time has gone on it became unwieldy as more people started to blog, some have left and blog names have changed. I used to list them with names I would remember but it started to become difficult to compare my feeds with that list.

I have spent some time and revamped the lists so that the names are the same as the blog name. This makes it easier for me to check that the list is up to date with my RSS reader. I have also demoted blogs that have not been updated in a while to the bottom category – old blogs. But there are still some great posts in these. Perhaps someone should pull together all the good posts into a mega document so that they do not get lost.

The list is split into Microsoft and community bloggers plus I have a list for SCE and another for community bloggers who do good posts about SCOM but are not exclusive. This allows me to include Techlog amongst others now.

I named mine starting with my name first. In retrospect that was a good idea and wish that others did the same. Having a list in my reader that all start with System Center then gets cut off does not help! Be proud of your blog and put your name at the front so it is easy to find in a list.

I have also starred feeds that I personally think are useful to add. Sorry to anyone who is not on the list but it helps narrow it down for newcomers. I personally read every blog on that list and if you are keen on SCOM then you should too.

Reporting Database

While looking at the database sizes in the new Operations Manager 2007 Performance and Scalability Guide it struck me how large the reporting database could become. More so as I am working with a client on MOM 2005 at the moment to try and tame the reporting database which is up to 500 GB.

This database had the Summary Reporting MP installed so it was aggregating the data but also keeping the raw data. So it was a case of changing the grooming parameters and dropping them down slowly so as not to knock out temdb and impinge on other jobs. A good pace to start is Pete’s post which then links off to good stuff by Marcus Oh on the DTS task and grooming and summary reporting.

http://www.it-jedi.net/2006/08/thoughts-on-summary-report_115689315829015304.html

Marry that with Kevin Holman’s MOM 2005 SQL queries and you can sort it all out. But slowly.

With 2007 the aggregation happens automatically so you don’t have to worry about it (so much anyway). And the report on the DW is superb. When I first saw it I thought “why can’t more reports be like this” and secondly “how did they do it?”.

PDF – Data Warehouse Properties Sample Report

Of more concern to me was what value the reports will give the organisation given that they may have to have a database of a few terabytes (plus the capacity to back it up). Organisations have mixed feelings about reports. Some think they may be useful but are more concerned with alerting. Managers seem to be keener on having the reports but are sometimes not sure what they actually want. And some people think it is a magic bucket that you can pull out details of any server over ant time period and give exactly what you are looking for whatever that may be. If only.

If you are not running the reports or even if you are running them but not doing anything with them why do you need reporting especially as it takes up a lot of space?

I would like to see an easy way for OpsMgr (the product group won’t go back to 2005) to show what counters are associated with each report (not having to scan through docs!). A tool that shows the list of reports with the associated counters and you just highlight the report you are not interested in and it creates an override for all those counters. It also knows that if the counter is used in another report it won’t override it until it is the last instance. No point in collecting something you will never use. Even better would be after you switched the rules off a dialogue asking if you would like to delete all those counters from the reporting database so it cleans it up for you without having to mess around with SQL yourself. After all you may not be sure you wanted that report until it collected data but then decided it was not needed. This would help tame the reporting database size. Next version perhaps?

Yet Another MOM 2005 Disk Space Script

A client wanted to check on disk space based on whether it was a local disk, in which case 15%, or a SAN attached disk, in which case they wanted warnings at 10GB. This made sense as the large disks attached to the SAN could have 100s of GBs free and still alert based on %. Unfortunately WMI can only see what is presented to the OS. Windows doesn’t “know” that a particular volume is on a SAN or a local disk. As far as it is concerned, the SAN volume IS a local disk, therefore WMI just sees it as another instance of Win32_LogicalDisk.

Not being a scripting expert I looked around and found a number of scripts done by others but none were exactly what I wanted. I looked at the modified storage state script by Gerald Mims – http://geraldmims.spaces.live.com/blog/cns!4EDF3C508D92E609!166.entry and the updated version by Scott Moss – http://www.myitforum.com/articles/2/view.asp?id=11442. I like the idea of the file that prevents the disk from being monitored (so added that) but did not fancy putting a text file on hundreds of servers for custom thresholds. With my lack of script skills I felt it was too complicated to change to what I wanted. I also looked at the one from Brian Wren which was nice and simple – http://www.faqshop.com/mom/default.htm?http://faqshop.com/mom/monitor/can%20mom%20rec%20cluster.htm. In the end I went with the one from Geeky Girl – http://mysehnsucht.spaces.live.com/blog/cns!759071229966AF5A!585.entry as it fitted best what I was trying to do and worked “as is” initially which always makes it easier. But shamelessly stole good ideas from the others.

The new script is designed to look at the size of the disk and dependant on size will either use % free space for small disks and GB free space for large disks. The client wanted a threshold of 15% for local disks and 10 GB for SAN disks. 10 GB is 15% if the disk size is 66.67 GB. Therefore I used a figure of 66 GB to be the dividing line to determine whether the disk was small or large but it is a parameter so you can chose your own cut off point.

Rather than a simple threshold a series of worsening thresholds was created as follows:-

Alert Level	Small Disk	Large Disk
Warning	15%	10 GB
Error	10%	5 GB
Critical Error	5%	1 GB

 I turned the hard coded figures into parameters and created the two logic chains depending on the size of disk parameter. I also borrowed the idea from Brian about using >0 to test (I was getting script errors from cluster servers without it) and heavily stole the description piece from the original storage state script which I then turned into sub routine to keep the main logic easier to read. The downside is that this is not a state based script but on the other hand it is easy to customise with the parameters, exclude drives with a file called NoDiskAlerts.MOM in the root of the drive.

I have tried to document the script as much as possible. Create a new script and copy this one in and create a timed rule to run the script and add the 7 parameters. Then create event rules for each level. All are described in the script.

custom-check-disk-space-based-on-size
Note that this ends in doc as that was the only tyoe of file I could upload. It is a text file.
Also thanks to Stuart for pointing out the INT() function.

 

Additional Community MPs

I have come across some more MPs.

Raphael Burri (Zurich, Switzerland) monitors Swisscom’s MS-OS based installations including IPTV product Bluewin TV which is using Microsoft’s Mediaroom solution – hence the media services MP.

http://rburri.wordpress.com/management-packs/ 

Microsoft – Active Directory
OpsMgr ‘07
Custom AD topology discovery MP
enable multi forest discovery
Jan, 15 2008

Microsoft – Windows Media Services 9
OpsMgr ‘07
Windows Media Services MP
for Windows 2003 only
Mar, 18 2008

Microsoft – Windows Scheduled Task
OpsMgr ‘07
Windows Scheduled Task MP
Feb, 14 2008

Raphael says – “I just learnt that Maarten Goet of Inovativ has published a very similar management pack a few days ago. The main differences are that Maarten’s MP is introducing an additional object type ‘Scheduler’ (if I understand correctly those are essentially Windows computers with Scheduled Tasks configured) and that he’s decided to look at the ‘Last Run’ property instead of the ‘Next Run’. Also Maarten’s features cool icons for diagram views. It’ll be up to you to decide which one you find more helpful.”

Plus a MOM 2005 one from Raphael.

Microsoft – MOM 2005 DWH
MOM 2005
MOM Datawarehouse MP
keep track of DTS job duration
Jan, 9 2008

There are three kinds of lies: lies, damned lies, and statistics."

A tale of caution.

I was called back into a customer where I had installed MOM 2005 and the problem was that there were “too many alerts” but no-one could say how many alerts were too many. There was a figure being bandied about saying that 1,000 service desks tickets were being generated by MOM each day and senior people were getting annoyed. I was surprised at the figure as I had done quite a stringent tuning job when I put MOM. So I was wondering what had changed.

I accessed the MOM console and started creating views to show me what was going on. And after a while I thought that MOM was fine and there were only 1 or 2 alerts per hour being generated. They did say that things had settled down a bit as they had fixed some infrastructure issues. There was case one. Rather than saying that they were getting alerts that pointed them to problems that they fixed and so MOM was good they were just complaining how noisy it was. Well it will be noisy if there are lots of problems!

I looked at what rules had changed (search for rules based on date last modified) and found not many changes. There were a couple of noisy rules for Project Server and some disc space rules had been copied but the originals had not been disabled. Some fine tuning was needed but it did not seem that there were that many alerts and most of the ones in the console looked like problems to fix and as they were repeating they were not generating any new tickets.

However that was not the end of it and I did a bit of digging. This MOM system has a simple script that takes alerts of warnings and above and passes them as events onto the 2 management servers and an NSM system picks them up and transfers them to the service desk. I went down to analysing the number of these events created (the event log filter is very hand for this as I can put in the event number and a date and it tells me the number of events that match). The figure matched up with what I was seeing in the console. Puzzling.

I received a few of the alert analysis reports that had been previously run and they were indeed showing over 1000 alerts per day. I picked a day and in the alert analysis report there were a large number of alerts but a quick look at them showed a particular pattern. I created a view that showed all alerts for that day (including resolved) and my suspicions were confirmed. Although it looked like there had only been 45 alerts created that day (according to the specific events in the event log) there were 710 raw alerts. 47 were information so they do count as they do not create tickets. There were however 594 alerts from 1 server that was in maintenance mode. There was the culprit. As each alert comes in it gets auto resolved as the server is in maintenance mode but the MOM server keeps the alerts in the database. So when the alert analysis report is run all these alerts show up. As each alert is auto resolved then none of them get to repeat so it looks like there were a lot of individual alerts. Minus the 24 synthetic events that tested the link once per hour and the numbers added up. Well one problem solved.

This shows that with these reports you can not take these things at face value and have to understand what is going on and where these numbers are coming from. Unfortunately the person that did the report did not and the figure of over a 1000 alerts a day started being bandied about. Nobody questioned it as it was in black and white in a report (so it must be true) and there was a perception in a part of the organisation that MOM was noisy and that was the problem. In fact one of the team leaders said that to me before I had even installed MOM! Talk about prejudging.

And it does not end there. The Unicenter NSM system that was the conduit to the service desk software had been doing its stuff as well. Once I received a report from NSM I could see a whole load of spurious alerts that had not been generated by MOM. In fact they were tripling the tickets being created. Apparently they decided to “helpfully” add some monitoring of their own but not tell anyone.As no-one knew that this was happening these alerts were not tuned. I was told by one of my old bosses years ago that during a project you can never over communicate and many times he has been proved right. Once the ticket was created it looked like it came from MOM and because of the perception about MOM being noisy it was accepted. We got them to stop that and now the number of tickets per day looks good. And most worrying no-one had thought to look at what was being created as tickets compared to the MOM console and alerts being generated. I was not given access to the service desk console as the cost per console was too high!

Lessons learned

1. Don’t believe all you read just because it is a report – dig in to find where the numbers come from
2. Get the story from both sides
3. Communicate
4. Promote MOM internally as a fixer of problems
5. Put down stories about MOM being noisy

Without a MOM champion MOM was getting a bad reputation. Numbers were being taken at face value without a detailed analysis of where they had come from.  It was only because I was adamant that MOM was working well that the investigation expanded. A sad tale of people too quick to blame the tool and relying on numbers without understanding them. MOM is a great tool in the right hands. Don’t dis it.

MP Assumptions

You would think that by now the major management packs from Microsoft would be settled, especially as many have them have been through a few updates over 6 years. It would be an assumption that many people installing MOM or SCOM for the first time would make – especially based on the marketing. But a number of posts recently made me think about how untrue that is.

Kerrie and Cameron post about how CPU % utilisation monitors don’t work as the Windows 2003 process class discovery is disabled by default. This is bizarre as every customer I have talked to wants to know about CPU utilisation. KB 948097.

Blake Mengotto, Discuss It Now mentions ongoing issues with the Exchange 2003 and 2007 MPs with MOM 2005. As this MP has been around for a long time and one of the major sellers of MOM and SCOM you would think it would be perfect by now. As anyone who has tried to use the wizard to do the advanced front end monitoring knows this is not so. And in 2007 the Exchange Back End server can not be detected as such if it is clustered. KB 948095. Worked OK in 2005.

Pete writes about how some baseline monitors in the Exchange MP are not appropriate.

Pontus Blomqvist mentions that the new SQL MP now has the SQL 2000 Blocking Analysis as active but alerting is disabled! You win some you lose some.

If you are using the console to keep track of alerts you could be in for a surprise as 2007 will groom alerts that are not resolved. See Pontus’s post. Also Kevin mentions that failed tasks are not groomed. And that in the OM MP the MP Notifier rule is not working.

If you think that MOM 2005 AD/DNS MP did things that 2007 does not do you would be right. Fortunately Matt has created an extended custom MP.

So basically you can not assume that any MP is 100% fault free. It would be nice to have full documentation by the MP creators of the FULL contents of their MP and not just an overview in the MP guides. I would like to see all rules/monitors listed and what they are trying to monitor and what assumptions have been made. Sure I can export the MP to XML and try and study it but I should not have to.

Windows 2008 Support Plans

On the 7th I posted that I would like to see what is going to be supported for MOM 2005 and SCOM 2007 with Windows 2008 and SQL 2008. Well Satya (aka Deployment Man from the MOM Product Team) has done a post about Windows 2008.

http://blogs.technet.com/momteam/archive/2008/01/09/support-plans-for-opsmgr-2007-sce-and-mom-2005-running-on-windows-server-2008.aspx

Basically don’t bother with MOM 2005 on 2008 except for agents but all roles on 2007 will be supported 90 days after RTM. And SCE will also be done but not for Core except for agents again.

Thanks for posting Satya. I wish all my requests were answered so quickly!

Now who is looking at support for SQL 2008 for these products?