Ben’s Tech Blog

We recently had hundreds of users leave our organization due to a takeover and we needed to remove each user from all of their distribution lists and security groups. Although we wanted to remove the users from all groups, I wanted to ensure that we had a record of what groups they were members of in case we needed to add them back in. The script that I created removes the user from all of their groups, but first records the list of groups they are member of and exports the list into a report file. We’ve already had two cases where we disabled a user and removed them from all groups and was later informed that the user’s termination date had been moved forward so their account needed to be reinstated. The user’s previous group membership listed in the report file made it very easy for us to add them back into the groups that they should belong to while they are still with the company.

I’ve added the script here.

http://news.bbc.co.uk/2/hi/technology/5243862.stm

An interesting summary of the development of the internet over the last 15 years. I finished school and entered the IT industry in 1995/1996 when the internet really began to take off. It’s amazing how the quickly it has evolved. I guess it’s similar to television for my parents’ generation.

Despite being an excellent product, Microsoft Exchange has some limitations when it comes to email management for teams. There are several approaches to implementing a team email delivery/management solution with an ‘out of the box’ Exchange deployment but unfortunately each solution has a drawback.

This topic has been on my mind a fair bit recently as we have been setting up team messaging solutions under a new company structure. In the past I’ve implemented custom form solutions to manage team messages so that all of the desired features are incorporated, however custom forms are not used by this company and they don’t wish to use them moving forward. With that in mind, we had to look at the default solutions and compare their behaviors before making recommendations to the various teams.

The following table illustrates the solutions that we offered to each team with the behavior of the replies to messages that are sent to the team. Interestingly enough, accessing a shared mailbox via OWA was often the best solution for customer service teams that wanted to be able to track their sent items.

When decommissioning Exchange servers, I like to stop the Exchange services or shut the server down for a week or so before uninstalling Exchange from the server; that way if I have overlooked anything and an issue arises I can bring the server back online very quickly. This week I was planning the removal of an Exchange 2003 public folder server. I re-homed all of the public folders and waited until they had all disappeared from the public folder instances container for the pubic store on the server in question. The next day I shut the server down and after a while noticed that there was mail queuing on the front-end servers to the public folder server that I had shut down.

At first, it didn’t make sense to me why the mail was queuing for the public folder server that I had shut down - there were no public folder replicas on the server, so in my mind there was no reason for any mail to be delivered to that server at all. After closer inspection of the messages in the queues, I could see that the messages were queuing for public folders that never had replicas on the server in question, which led me to believe that the server was playing some type of ‘bridgehead’ role that I didn’t know about.

I came across this Technet article that explains how public folder routing works and it cleared up the confusion that I had. In summary, when a message is received by a front-end server (or any Exchange server that doesn’t contain a public store), the server queries Active Directory for a public folder server that contains a copy of the public folder hierarchy. The transport then delivers the message to the server that contains the hierarchy based on the result of the AD query. The server with the copy of the hierarchy then determines which server or servers contain a replica of the public folder that the message is addressed to and passes it on to the appropriate public folder server containing the replica.

As the article explains, the query result will always be the most recently deployed public folder server for the routing group. This was consistent with the issue that I had – the server that I shut down was in fact the most recently commissioned public folder server, so it made sense why the messages were queuing up. Of course, I powered the server back on to allow the messages to deliver, and then began to look at how I should approach the situation.

Uninstalling Exchange would had been the easiest fix, but I still wanted to keep the server offline for a few days before uninstalling Exchange. The article mentions that you can modify the msExchOwningPFTree attribute on the public store object, and this is the solution that I decided to go with:

“Editing the msExchOwningPFTreeBL attribute directly is not possible, because it is a back link. However, you can delete the msExchOwningPFTree attribute on a specific public store object. If you delete the msExchOwningPFTree attribute on the public store object that is unavailable, the msExchOwningPFTreeBL back link attribute will be automatically removed from the back link list. By changing the msExchOwningPFTree attribute, you change the list in the msExchOwningPFTreeBL list, and thus alter the public folder routing decision.”

The article doesn’t explain how to atually edit the attribute so I thought I’d post the solution here so that others could refer to it. Note that after I cleared the attribute I wasn’t able to mount the store without putting the attribute back in place again. Also, you can avoid this scenario by mounting an empty public folder store on each front end server. The solution below however provides an easy way overcome a mail routing issue similar to the scenario that I had, particularly if you have already shut down the server and can’t bring it back up immediately.

If you want to temporarily prevent a particular public store from being used as the initial delivery point for messages destined for public folders in a routing group, you need to clear the msExchOwningPFTree attribute on the public folder store object using ADSI Edit:

• Launch mmc.exe
• Add the ADSI Edit snap-in to the console
• Right click on the ADSI Edit container and click on Connect to
• Choose the Select a well known Naming Context option and select Configuration from the drop-down list, then click on OK.
• Expand the Configuration container, then expand out each container as follows: CN=Services then CN=Microsoft Exchange then CN=YourExchangeOrg then CN=Administrative Groups then CN=AdminGroupofyourPFServer then CN=Servers then CN=YourPFServer then CN=InformationStore and then click on CN=StorageGroupContainingYourPFStore.  In the pane on the right you see the public store object listed.  Right click on the object and click on properties. 

  

• In the list of attributes, double click on the msExchOwningPFTree attribute and then click on Clear and and then OK

  . 

  

Once I cleared the msExchOwningPFTree attribrute I shut the server down again.  I sent some test messages and monitored the queues on the front-end servers.  The messages were delivered to the public folders, and no messages queued on the front-end servers. When it came time to remove Exchange, I put the attribute back in place so that I could mount the public store, take a final backup and uninstall Exchange.

I’ve added 2 scripts that I created recently to the scripts page. Check out the links below.

VBScript - Export a list of Mail Enabled Public Folders visible in the GAL
VBScript - Export a list of all Mailboxes with their home Server, Storage Group and Database.

We had a user contact us yesterday who had deleted a public folder that contained a large number of subfolders and items. This is not usually too much of a drama, we have a 14 day deleted items retention period on our public folder stores, so its usually just a case of adding the DumpsterAlwaysOn key to the registry on the client, browsing to the parent public folder, clicking on Tools, then Recover Deleted Items and selecting the folder to recover. This time however, we received the following message when attempting to recover the folder:

“Outlook was unable to recover some or all of the items in this folder. Make sure you have the required permissions to recover items in this folder, and try again. If the problem persists, contact your administrator.”

Naturally, we checked the permissions; in fact we tried to use the Exchange service account mailbox to recover the folder but received the same error. We also tried temporarily disabling Trend ScanMail (our Exchange level antivirus) in case it was interfering with the recovery process. The folder would actually appear in Outlook temporailly before it produced the error, at which point it would disappear again.

We did a lot of searching around to find a solution; most forums and articles that we came across either suggested to check permissions and disable the antivirus like we had done or use isinteg to repair the database (which I’m not such a fan of). Others suggested that it wasn’t possible to recover the folder at all. After some more searching I came across a tool called MFCMapi.

Once I downloaded MFCMapi, I was able to browse the folders that had been deleted. I tried to recover the folder that the user had deleted, but it produced an error. However, because I was able to browse the sub folders, I attempted to recover one of the them and it was successful. I tried several folders, some were successful and others were not. I figured if we could get some of the contents back it would be better than nothing. It appeared that some of the contents were corrupted, which would explain was why Outlook was unable to recover the folder. Outlook doesn’t provide the option to recover the sub folders, and this is where MFCMapi really helped. I was able to restore the majority of sub folders with 5 or 6 of them not wanting co-operate.

You can download MFCMapi from here.

Once downloaded and extracted, run MFCMapi.exe and close the welcome window. To browse deleted public folders do the following:

1. Click on the Session menu and click on Logon and Display Store tables. You’ll see that it uses Outlook to connect to the stores, select your Outlook profile and click on OK

2. Click on the MDB menu and select Open Public Folder store. Tick the Use Admin Privilegesbox and click on OK.

3. Expand Public Root and IPM_SUBTREE. You’ll then see the public folders that you would normally see under “All Public Folders” in Outlook.

4. Navigate through the public folder tree until you get to the parent folder of the folder that you are trying to restore.

5. Right click on the parent folder and select Open Deleted Sub Folder Table. A new window will open displaying the deleted subfolders. Right click on the folder and select Restore Deleted Folder. A box will open up that allows you to select to also restore the subfolders, tick that option and click on OK. You may receive an error message stating that the folder couldn’t be restored, much like I did.

If you don’t receive the message then the restore was successful and you should see the public folder and items appear in Outlook. If not, don’t panic just yet, you may be able to restore some (hopefully most) of the subfolders. If this is the case, just navigate through the folder structure and attempt to restore each folder. Unfortunately it doesn’t allow you to select multiple folders using CTRL or SHIFT; I spent a fair bit of time restoring each folder one by one.

Considering that I was able to restore most of the folders and items that had been deleted, I was happy with the result, and so was the user. MFCMapi proved to be a very useful tool and I’m sure there will be times when I’ll need to use it again.

We have been running Cisco Call Manager for some time now as well as the Cisco Attendant Console software. Recently, the users of the company that we have merged with moved in to our office and we rolled out Attendant Console to the new management team. One of the managers reported that the line status was not updating (the icons that indicated whether or not another user was on the phone), instead they were displaying question marks.

After a bit of troubleshooting, I discovered that the Attendant Console software registers the client’s IP address with the Call Manager server; I’m guessing that the server uses this IP address to send line status updates back to the client. You can view the IP address that Attendant Console has picked up by openning the Advanced tab in the Attendant Console Settings dialog; it displays the IP address and port where it says “Local Host IP Address (for line state)”.

When I checked the Local Host IP Address I discovered that it had picked up an APIPA address (169.254.x.x). I then did an ipconfig and saw that this machine also had a bluetooth adaptor with the address that was registered in Attendant Console. Fortunately, when I asked the manager about the bluetooth adaptor he said that he didn’t use it anymore so I just disabled it. Once I disabled the device and restarted Attendant Console I could see that it had registered the correct IP address and the line status was updating as normal.

I’ve had a look at the documentation but there doesn’t appear to be a way to force Attendant Console to use a specific adaptor. You can manually specify an IP address to use, but in most environments where DHCP is used this is not practical.

It’s been a little while since I’ve udpated my blog, mainly becuase I’ve been busy with work and studying. I passed the Exchange admin exam yesterday (70-284 - Implementing and Managing Microsoft Exchange Server 2003) and I’ve got a few days to breath before I start on my next exam: 70-293 - Planning and Maintaining a Microsoft Windows Server 2003 Network Infrastructure. As usual, I’ve created a Mind Manager map to create notes against:

70-293 Objectives Mind Map

Passing the 70-284 exam has given me the MCSA:Messaging status, and I have 2 more exams to take to complete the MCSE. Free time? What’s that?

Dealing with Member Servers

Re-deploying services

In some cases, rather than migrating a server from an NT4 domain, it made more sense to transfer the server’s function to another server or device. For example, DHCP was running on the PDC of one of the legacy domains to issue IP address leases to the workstations in their subnet. I opted to move the function to our switches. We are using 2 Cisco Catalyst 4507 switches; I worked with our network services team (the Cisco guys) to move the scopes from the NT4 server to the 2 switches. Each switch was given it’s own pool of IP addresses for redundancy and load-balancing.

Migrating Member Servers

There were several servers which needed to remain in use. For these servers the migration from their existing NT4 domain to the AD domain was as simple as removing them from the NT4 domain and adding them to the AD domain. The service accounts and ACLs on objects were already using groups from the AD domain (there were external trusts in place between the AD domain and the NT4 domains), and when I added the servers to AD users were still able to access resources on the servers. I tested this process in a lab before migrating the actual production servers.

Decommissioning the actual Domain

Removing the last Domain Controllers

I have completely decommissioned 1 domain so far, with the remaing 2 domains almost done. My aim was to remove all member servers until I was down to a PDC and a BDC. Once I got to that point, I disabled all of the user accounts except for the Administrator and REPL accounts. After a fews days, I then removed the trust relationships.

I took a backup of the PDC, but I wanted to have a very easy way to bring the domain back up if I needed to in future. I considered relying on a Ghost image, but that would require that the image be dumped back to the same hardware, and I knew that the hardware would be scrapped in the near future, leaving the image useless. Although I had the standard backup of the OS, I didn’t want to have to go through the whole restore process if for whatever reason I needed to bring a domain controller back up. My solution was to build an NT4 server in a Virtual PC image and promote it to a BDC in the legacy domain. Once I confirmed that the database had replicated, I shut down the domain controllers and burnt the Virtual PC image to DVD. If I need to bring the domain back up in future, it will be a simple case of starting the VPC session.

As a network admin, I’m often handed software and asked to deploy it for a bunch of users. In a lot of cases the software is provided by a supplier or customer and in some cases is used to connect to their server over the internet. I’m amazed at how much of this software is, in my opinion, poorly written. Here are a few things that make me cringe, and often give me a headache.

Local admin rights

Running as a non-admin is obviously a best practice security measure, but a lot of developers make it difficult to put it into practice. When we went through the process of removing local admin rights from users, we ended up with a fairly long list of software that wouldn’t work without admin rights. To get around this, we used the Sysinternals Regmon and Filemon utilities to find out which registry and file system objects were giving us “access denied” errors. After identifying these, we were able to modify the object’s permissions to give the local users group the required access.

I come across a lot of software that doesn’t work without elevated user privileges, and I often have to go the same process as above to get them to work under an ordinary user account.

No Proxy support

We are running several pieces of software that cannot be natively proxied. By that I mean that they need to connect to servers using ports that are not sent to a proxy server, even if one is specified in IE. Sometimes we are able to get around this by installing the Microsoft Firewall Client (an ISA feature), but at the moment I am testing software for users who are required to use a Squid proxy, and I’m not aware of any equivalent solution.

There have been cases where using the firewall client doesn’t work either, so opening the required ports on the internet firewall is the only option. This is a relatively simple solution if you have a local internet connection, however a lot of companies (like ours) only have internet connections in regional centers which makes it more difficult. In situations where the software is installed in a remote site, we’ve managed to implement a work-around by creating a translation on the local router (for the network that the user belongs to) to convert the public IP address (that the software attempts to connect to), to a private IP address (that belongs in the subnet that the internet firewall resides in). Once that translation is in place, another translation is required on the internet firewall to translate from the private IP address to the original public IP address. Although it’s possible to do this, it’s not ideal, and in our case, involves a third-party (the WAN provider).

Java

It may be a nice development platform, but running multiple java applications can be difficult, particularly when they use different versions of Java. I’m currently testing a desktop environment with 5 different Java applications; only 2 of these applications use the same java version, so there are 4 different versions of Java installed. Java doesn’t seem to be completely backwards compatible, so if an application is developed for Java version 4, it may not work if only Java version 5 is installed on the workstation. In order to get the application to work, you need to install Java version 4. The problem is, to install Java version 4, you first need to uninstall version 5, then install version 4, and then of course install version 5 again so that your other Java app will work. I spent a fair bit of time trying to get all of the Java apps working in harmony. The golden rule it seems is to install the oldest version of java that you require first, followed by the next oldest version and so on. In any case, its a head-ache, so I never like receiving software written for Java.

I’m sure if I thought harder I would be able to list more things that annoy me but these are the three things that have caused me headaches in the last few days.