Big Scripts

While working on a MOM 2005 system all the servers running DFS went red. This organisation had set these up but had then disabled them all temporarily for security reasons and have not re-enabled them.

Looking at the cause there is one big script that runs every 15 minutes. This is the “DFS root, link, and target availability health check” script. There is no comments in the script to help. As I am not a script specialist it is hard to work out what to do to remove the bits that cause the alerts that are not needed while still providing monitoring of the other bits. In the end, due to time constraints, I disabled the whole script from running and the organisation will switch it back on when they enable their DFS shares.

There are other big scripts that I have come across in the past that are similar. So why do the product groups feel the need to write these huge multi function scripts? Because they are clever? They want to show off? Or they feel they can reuse code better?

I would prefer to have those scripts broken down into smaller bits that do just one thing. That way it would be easier to just switch that bit off. Tuning is a crucial part of a MOM or SCOM installation. Operators will get blase if the console is full of false alerts and the tool will fall into disuse. This end game needs to be in mind when the product groups are creating their MPs. And with the difficulty of applying overrides in some areas of SCOM 2007 it does not look like that message has got through. If they need help they just have to hire me. 

Exchange MP Request to Product Group

From the early days I have always though that the Exchange MP was a great example of using MOM to the full. They put in cluster awareness when MOM 2000 was not! They put in a lot of advanced features but unfortunately to get these features working you had to configure them.

In MOM 2000 you had to manually configure the registry settings. In 2005 they had the MP Wizard which helped a bit but was confusing and needed full Exchange and Domain rights to run it successfully. In SCOM 2007 you still need the wizard. And you can see the problems by the fact that Microsoft had to release a white paper with the top three issues.

With the 2007 MP as soon as you install the MP you still get alerts from the bits that need manual configuration just like previous versions. And the scripts to use are not as robust as they should be!

My recommendation to the product group is to split the MP into two parts. The bulk part which you can just import in and it just works without configuration. There are hundreds of rules that would work out of the box.

Then there would be a second MP which would contain all the bits that needs manual configuration or else it will create alerts. That way the bulk monitoring can be imported immediately and once settled in then the administrator can then decide when to install the advanced MP when they have time to configure it. That can’t be too hard can it?

Exchange 2007 MP

A few people have written about importing the MP into 2007 and some of the issues.
http://contoso.se/blog/?p=218
http://ops-mgr.spaces.live.com/Blog/cns!3D3B8489FCAA9B51!240.entry plus some trouble shooting.
And a problem with it on MOM 2005 at http://discussitnow.spaces.live.com/Blog/cns!A4408C121568CAA4!1631.entry if you are also running Exchange 2003.

I have been busy installing it with MOM 2005 ( a joy to work with again after SCOM 2007) as the organisation wants a fully supported solution. I have heard a rumour that the MP might be released in October for 2007. My view is that they are working on the MP in parallel with Exchange 2007 SP1. That would fit in with how the product groups work. Interestingly in some of the alerts when you go to the web site (there is limited info in the Knowledge tab and everything seems to point to the web – shape of things to come – not good if you have to run the server in a secure environment with no Internet access as some government agencies need to do) the articles mention SCOM 2007 as well as MOM 2005. First time the documentation has been written before the product!

Having read the MP guide I though that this would be better as there is no longer the Exchange MP Wizard – hurrah! But there are 2 manual configurations but it is done by PowerShell scripts already supplied on the Exchange server. Should be easy! But only if it works first time. The MP guide is a whopping 72 pages but the MOM MP bit does not start until page 42.

To monitor OWA, Active Synch and Web Services Connectivity then you need to run New-TestCasConnectivityUser.ps1 on the Exchange servers. And just like the old wizard you need to have Exchange and AD rights as it creates a user for each mailbox server named CAS_{GUID} where the GUID is the first half of the GUID for each mailbox server. If all the servers are in the same domain then the one script will do them all by piping in the mailbox servers (get-mailboxServer | .\new-TestCasConnectivityUser.ps1). If they are not you need to log into the servers in the other domain with the correct rights.

The MP Guide talks about running “set-owavirtualdirectory” but the organistion already had it set. The guide assumes that you have not set it up – or so I assume. They don’t mention what to do if you have set up OWA already. They should provide a test to check that it is OK.

That is the problem with some of the guides. They are very long but light in the area where you need the most help. This bit is just 1 page. I know someone from Microsoft UK who went to Redmond a few years ago to help write a white paper for a few weeks. The story about the process and how it was done was not encouraging.

Although that was set up the alerts still came up and the alert mentioned running the set-CasConnectivityCredentials script. But that script does not exist. Although Lee Owens has investigated the issue and found that if he ran the first script again it sorted things out. We ran the script again and this time it worked and gave the user which was previously created the correct rights. Scary stuff from the Exchange team.

Another weird thing is that the Exchange MP complains about running SP2 on Windows!
Rule Path – Microsoft Exchange Server\Exchange 2007\Common Components\Best Practices Analyzer\
Rule – Pre-release software detected. – event 1875
Apparently the built in BPA can only deal with servers on SP1.

There is a good three part guide by Brazilian consultant Anderson Patricio that covers setting up the Exchange 2007 MP including MOM 2005 parts and even configuring an ISA 2006 server in a DMZ that is publishing Exchange. For people that know MOM some chunks will be a bit obvious but it is a good guide non the less. He provides a list of all cmdlets that MOM uses to gather information from the Exchange in part one. Very useful as the official guide does not mention them!

Monitoring Exchange Server 2007 using MOM 2005 (Part 1)
Monitoring Exchange Server 2007 using MOM 2005 (Part 2) Monitoring Exchange Server 2007 using MOM 2005 (Part 3)

Also useful
http://www.microsoft.com/technet/serviceproviders/hmc4/CMSU_HE_DW_PROC_Install_and_Configure_Exchange_Monitoring_and_Data_Collection.mspx?mfr=true

Can’t wait for the version for 2007. Hope they fix the bugs.

Blueprint for a Big SCOM Design

According to the testing done a single management group can support up to 5,000 agents. If those are all servers then there will not be that many organisations that will need multiple management groups and those that do will be big enough to hire specialist consultants!

From the Design, Guide, Deployment Guide, Security Guide, Performance and Scalability guide I have come up with what I consider to be the blueprint design to put in place for Operations Manager to monitor from 2,000 to 5,000 servers.  The first thing is that if that many servers are being monitored then I am assuming that you would want a fault tolerant design.

In 2007 both the database and the RMS server are single points of failure and so need to be clustered. It would be nice to have them on a single cluster but at the moment that is not a supported option although Microsoft may support that in future. So 2 separate clusters are needed.

To scale up you will want x64 for Windows for the database and RMS servers. 64 bit for SQL server although I have seen posts of people running 32 bit SQL on 64 bit Windows. After some discussions with some people from Microsoft IT and the Program Manager responsible for testing non RMS management servers can be 32 bit and have the potential to  be virtual. For performance you will want 8 to 16 GB of memory on both the RMS and SQL servers. You can start with 4 but the reason for going 64 bit is to use memory above 4 GB. Obviously database I/O is crucial to the performance and SAN disks have to be organised accordingly.

On the RMS server the SDK service is responsible for providing a communication layer between the OperationsManager database and the rest of the Management Group. The Config Service is responsible for calculating the configuration of all agents and which Management Packs they should receive and the overall configuration of the Management Group. To allow the RMS to deal with these tasks and all the security requests from consoles and web consoles which get funneled through it, it is recommended that no agents are assigned to the RMS. To deal with agents then three management servers are used which will provide the scale and fail over.

Finally I would keep the Data Warehouse on a  separate server (preferably x64) which does not need to be clustered as it is dealing with long term reports which are not usually deemed to be mission critical with one caveat. The management servers now insert data directly into the data warehouse database so if this is off line for a period of  time then some data may be lost.

For a small number of servers then SCOM 2007 on a single box will do. When you get into a few hundred servers to a few thousand then there are many decisions that need to be made and consulting an expert will help. But for 5,000 agents I think that this is a good blueprint.

Note that I have not put in all lines of communications as the diagram gets messy. Also as per a previous post I am treating the product as three separate products but with this in place then it is possible to add ACS collectors and databases as needed. As I am only looking at servers I have not included AEM. This also assumes all servers are in a single forest but for DMZ then a certificate server and perhaps a Gateway server need to be considered.

Agent Proxy

First there was the OpsMgr console GUI and it was bad. You can only set the agent proxy one server at a time! There was a wailing and gnashing of teeth. Especially from those people who had to do hundreds of AD and Exchange servers.

Then there was the PowerShell script. Crafted by a Redmond Program Manager and forged under the blinding white heat of the SystemCenterForum.org.  And it helped. Greatly.

Along comes OpsMgr support guru Clive Eastwood with ProxyCFG.exe. Easier to use that the script if you could get around the arcane requirements of the parameters.

Finally we have Boris Yanushpolsky’s Proxy Setting tool. It is a GUI. It is simple. It shows you the groups, members and the state of the proxy settings. You can change the settings per server or multiple servers at once. The world sighed. All was good. The GUI was restored back to glory. Thank you Boris.

And to help Cameron has created a list of MPs that need the setting and those that don’t. Very handy.

But why is the OpsMgr Console so bad. The team knew that all AD and Exchange servers at a minimum needed this setting. Is Microsoft so enamored by UNIX scripting that they think everything should be done in PowerShell? PowerrShell is a useful tool to have and I am sure many people will benefit from it. But we are Windows administrators brought up on the GUI. Microsoft – don’t turn your back on us. We want a good GUI. And I hope SP1 delivers considering it is not getting released until Feb 08.

OpsMgr Updates

A few people have already mentioned about the new updated MPs for Operations Manager itself. You will definitely need the backward compatibility MP in order to import some of the new MPS that have been converted from MOM 2005 and the others contain important fixes.

Here are what the new MPs contain (dates in US format) :-

Operations Manager 2007 Data Warehouse Library Management Pack
Version: 6.0.5000.26
Date Published: 8/22/2007
Microsoft.SystemCenter.DataWarehouse.Library.mp 6.0.5000.26

OpsMgr 2007 MOM 2005 Backward Compatibility MP
Version: 6.0.5000.12
Date Published: 7/12/2007
Microsoft.SystemCenter.Internal.mp 6.0.5000.16 ***
Microsoft.Systemcenter.2007.mp 6.0.5000.16 ***
System.Mom.BackwardCompatibility.Library.mp 6.0.5000.12

System Center Operations Manager 2007 Management Pack
Version: 6.0.5000.28
Date Published: 8/22/2007
Microsoft.SystemCenter.2007.mp 6.0.5000.28
Microsoft.SystemCenter.Internal.mp 6.0.5000.28
Microsoft.SystemCenter.Library.mp 6.0.5000.28
Microsoft.SystemCenter.OperationsManager.200.mp 6.0.5000.28
Microsoft.SystemCenter.ServiceDesigner.Library.mp 6.0.5000.28
Microsoft.SystemCenter.WebApplication.Library.mp 6.0.5000.28
System.Health.Library.mp 6.0.5000.28

As you can see you do not have to do 2 of the MPs that were released as part of the Compatibility MP. But it does not make any difference if you do as the new versions will overwrite them. And if you do them in the other order OpsMgr will warn you. A very useful new feature for hard pressed admins.

You can check which MPs are installed and their version number in Administration, Management Packs but you can also run a report that shows that information if you want a printout.