Force uninstall of Windows Server Update Services (Server 2008 RTM)

Last week, we had a hardware failure that cause corruption of one of our SCCM site servers.  To clean in up, I needed to uninstall WSUS from the server.  Unfortunately, attempting to remove the role only gave me an error message saying removal failed.  After pieceing together stuff from the web, I think I now have a how-to when it comes to manually remove WSUS.  Since this is the second time I’ve had to do it, I thought it best to write it down this go-around….

  1. First, determine the application’s product code GUID.  You can do this by looking in the registry at \\HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall.  You will need to cycle through the subkeys until you find the one with the correct name in the DisplayName value.  Make note of the subkey; that is the GUID.  To save you the trouble (in case you’re working with the same version as me), the GUID for WSUS 3.0 SP2 is {2C0D7E35-EE6E-4DC7-BA13-2C68AEDEB59D}
  2. Now, at an administrator command prompt, using MSIZAP.exe (which is included in the Windows SDK), clear the install entry:
  3. Stop the WSUS services:
    net stop wsusservice
    net stop wsuscertserver
    NOTE: If your WSUS install is truly broke, you may just get message saying the services are not online.  In that case, just proceed to the next step.
  4. Delete the WSUS services:
    sc delete wsusservice
    sc delete wsuscertserver
  5. Detele the Windows Internal Database:
    msiexec /x {CEB5780F-1A70-44A9-850F-DE6C4F6AA8FB} callerid=ocsetup.exe
  6. Delete the WSUS program folder: C:\Program Files\Update Service (this may vary if you’re using an x64 OS)
  7. Open IIS manager and delete the WSUS virtual directory
  8. The role will still show in server manager, but can be reinstalled using the WSUS installer package.

SCCM Cluster Site Server error 2147942467: The network name cannot be found

We had recently done an SCCM database site server migration, and this error started popping up:

SMS Site System Status Summarizer still cannot access storage object
"\\server\share$\SMS_server" on site system "\\server". The operating 
system reported error 2147942467: The network name cannot be found.

Upon further investigation,  I realized that I forgot to place NO_SMS_ON_DRIVE.SMS files on our clustered disks (see Microsoft KB 871234.  Because I failed to do that, SCCM installed its components on the disk with the largest amount of available space, which in this case, happened to be a shared database disk.  The error crops up once the clustered disk is moved to another node (since its no longer available on the machine in question).  To fix this error, Microsoft directs you to drop and re-add the site server, after placing the NO_SMS file on the appropriate drives.  But, for a database site server, this is not really an option (unless you happen to have another database available somewhere to make a temporary move).  So, I was now left in a situation where I had to fix it in place.

To do this, first make a copy of the SMS directory to it’s new location.  Then, change the path in the following registry locations:


On the primary site server, change the path in the following registry location:

Multisite Component Servers\<Target Server>\Installation Directory

Restart the Site Component Manager Service and the issue should resolve.

SCCM Query and Report Aliases

When crafting queries and reports in SCCM (or SMS), it is helpful to know that the WMI class and SQL server naming conventions are slightly different.  This page is most helpful in this respect (of course, it’s hidden deep in the bowels of Technet):

SCCM OOB Management and Intel AMT MEBx (vPro) custom certificate hashes

If you run the Delete Provisioning Data from Management Controller Memory command on a workstation in SCCM, not only does this unprovision the machine for OOB management, but any custom certificate hashes that you entered in MEBx will be deleted as well.  If you want to provision this machine again, you will need to go back in and re-add the hash.

Issuing certificates directly to smart cards with Windows Certificate Manager

We’ve been working on a plan to deploy smart cards on our standalone network.  For initial certificate issuance, we planned on using the certmgr.msc MMC snap-in.  This is done by right-clicking the Personal folder and selecting All Tasks | Advanced Operations | Enroll On Behalf Of… option.  Everything seemed to be working, but the certificate was only issuing into the local store, and not onto the smart card.  After some trial and error, we found that our issue was an incorrect CSP for the certificate template.  By default, the Smartcard Logon template allows for the use of any CSP (with the Base Crypto Provider as the default).  Prior to issuance, the certificate template must be customized to issue directly to the smart card.  First, make sure that the “Allow private key to be exported” option is NOT selected (otherwise, the next setting won’t be available, which was our problem).  Then, ensure the CSP is set so that the Microsoft Base Smart Card Crypto Provider (or whatever CSP your smart card uses) is the only one allowed.  Once the template is published this way, logon certificates will issue directly to the card!

SCCM 2007 SP2 Operating System Deployment and multi-tiered NICs

We’ve begun deploying Windows 7 with SCCM’s Operating System Deployment (OSD) capability.  We’ve found that some of our workstations use multi-tiered LOM NICs.  Turns out Windows PE HATES multi-tiered NIC drivers.  So, if you’re in this situation, you’ll need to get a monolithic driver for your Windows PE boot.  See for more information.

We’ve also noticed that if you use DHCP, be sure that your leases are long enough to cover the OSD sequence up to the first reboot.  If your lease expires before then, you’ll get cryptic errors (like 0x80072ee7) and your IP address will change to a link-local.  We had a DHCP range that was set to 5 minutes (!) and it was causing this failure… and we had a heck of a time trying to figure out the root cause!

SCCM 2007 SP2 vs. Windows 7 SP1 and Server 2008 R2 SP1

If you’re currently on SCCM 2007 SP2 and have started to deploy Service Pack 1 for Windows 7 and Server 2008 R2 SP1, be sure to grab the MS hotfix that provides support in SCCM for these new versions:

EDIT (5/19/2011): Once you install this hotfix, you’ll need to go back and verify any drivers or applications that have OS run restrictions configured.  Because this adds the SP1 operating system as a selection, it will fail to install on SP1 if it is not specifically checked.

Error 0x80092026 during Windows Updates

This error came up when we were trying to send out an update package to our Windows XP machines.  Microsoft has a couple of pages that describe what you can troubleshoot to fix this issue:

None of the suggestions worked for us.  Since our environment is locked-down via draconian Group Policy Objects, I thought that would be a logical place to continue troubleshooting.  After slumming around a bit, I discovered that Computer Configuration | Policies | Windows Settings | Security Settings | Public Key Policies/Certificate Path Validation Settings/Trusted Publishers/Trusted Publishers can be managed by: was set to All administrators only.  This has to be set to All administrators and users for Windows Update (SCCM, SMS) to work properly.

Don’t say I never helped you out…

PKI certificate autoenrollment fails on Windows 7

Today is a blogging double-feature!  If your network has any type of security hardening (e.g. FDCC, DISA STIG, etc.) you may end up in a situation where your Windows 7 and 2008 R2 machines are not autoenrolling for PKI certificates from your internal enterprise CA.  We found that the following two settings must be enabled in policy:

1. NETWORK SERVICE must be given the “Access this computer from network” user right.  This allows the machine to enroll certificates.
2. The “Task Scheduler” service on the target machine must not be disabled.  The Certificate Services Client uses this service to autoenroll PKI certs.

WinPE SCCM boot image or OSD task sequence failure due to bad NIC driver

Have you experienced a problem loading network drivers into your Windows PE boot image on SCCM, or does your SCCM OSD task fail partway during OS install?  Well, read on!

If you get an error like “Failed to inject a Config Mgr driver into the mounted WIM” or an 0x80070040 “Access Denied” during OSD installation, it is due to the fact that WinPE needs monolithic NIC drivers.  You can verify this if you pull up a command prompt in WinPE (if enabled) by pressing F8 and running an IP config or ping check.  You won’t be able to contact any other machines and/or you’re assigned a link-local address.  For possible fixes, see:

Note that if you load a multi-tiered driver at ANY TIME during the OS install before the machine does its first reboot, that driver will take over and cause the connection to fail.  To work around this, create two driver packages, one without the NIC driver, that is applied during the WinPE phase and another with the NIC driver only, to be applied after reboot.

SCOM 2007 Logical Disk Critical Warning

If you get a logical disk showing as “Not Available” in SCOM, that actually is online and (apparently) working, this is most likely a result of the dirty bit being set on the volume.  You can verify this by running fsutil dirty query <drive>: on the machine in question.

According to Steve, this is due to the fact that the LogicalDiskHealthCheck.vbs script looks for this property when it is run.  Too bad it flags this as a critical error, when it really isn’t.

To fix this, run chkdsk /f /r <drive>:. Note that if this error occurs on a boot volume, the checkdisk has to be run at reboot.  Once the check is complete, you can recalculate the health on the monitor and it should reset. Also, it wouldn’t be a bad idea to put this in your SCOM knowledge while you’re at it…

Internet Explorer Sucks

I know that this post deviates from my normal techno-babble, but I was so annoyed/amused by this I couldn’t resist.  I normally use Internet Explorer, mostly because I’m too lazy to bother with anything else.  But today, when browsing Microsoft Support, IE served me this:

Just to make sure that the site itself wasn’t honked, I took a look in Chrome:

I don’t know about you.  But if I was a company that had its own web browser, I would at least make sure that my own websites could be properly rendered on them!

A sample 802.1X configuration guide

Due to the complexity of 802.1X configuration, there is a lot of documentation out there. Unfortunately, due to the complexity of 802.1X, there are very few step-by-step guides on actually setting a system up to use it. This is merely a crib sheet that I use to create a nominal 802.1X configuration. This setup uses computer certificates only, with users logging in with passwords (not smart cards). The servers are Enterprise 2008 R2, and the clients are Windows XP SP3 and Windows 7.  Network gear consisted of Cisco 3750 and 2960 switches running IOS 12.2(52).  Note that you have to have a K9 version of the IOS, otherwise the security features are not available.  These instructions may or may not work if you are on different server/client versions.  I know there are NPS client limitations on the Standard server version and that you cannot create custom certificates on Standard either.  So Enterprise or better is required for your servers.  Hopefully, someone out there will find this helpful. As there are many, many, many ways to configure this stuff, you may need to adjust it for your needs. I cannot profess that I am an expert by any means, but if you have a question about something in (or not in) this checklist, please leave a comment.

Configuration of 802.1X is a multi-step process. Reference: Intel vPro AMT, if used, should be configured and operational prior to running this procedure.

  1. Certificate Services
    This procedure assumes that an Enterprise CA has been previously installed and that client certificates have been provisioned to all workstations in the enterprise via Group Policy. Additionally, your primary site servers have been added into an AD security group called ‘SCCM Primary Site Servers’
    1. In this step, we will create and deploy the NPS certificate template.
      1. On the certification authority server, start the Certification Authority MMC snap-in
      2. Expand the CA node.
      3. In the console tree, right-click Certificate Templates. Select the Manage option. The Certificate Templates console will appear. Note the Domain Controller that the Console is connected to.
      4. In the details pane, click the RAS and IAS Server template.
      5. On the Action menu, click Duplicate Template. In the Duplicate Template dialog box, select the Windows Server 2003, Enterprise Edition template version.
      6. Click OK. The Properties dialog box for the certificate template opens.
      7. On the General tab, in Display Name, type a name for the new certificate template.
      8. Click the Security tab. In Group or user names, click RAS and IAS servers.
      9. In Permissions for RAS and IAS servers, under Allow, select the Enroll and Autoenroll permission check boxes, and then click OK.
      10. Close the Certificate Templates Console.
      11. Force replication across Domain Controllers using Active Directory Sites and Services. Begin replication from the server noted earlier.
      12. In the Certification Authority Console, right-click the Certificate Templates node. On the Action menu, point to New, and then click Certificate Template to Issue.
      13. Select the template that was just created, and then click OK.
    2. (If using Intel AMT, otherwise go to next section). Here, we will create the Client Authentication certificates for 802.1X AMT-Based computers. This is separate from the normal client certificates issued by the domain.
      1. In the console tree, right-click Certificate Templates. Select the Manage option. The Certificate Templates console will appear. Note the Domain Controller that the Console is connected to.
      2. In the details pane, right-click the entry that displays Workstation Authentication in the column Template Display Name, and then click Duplicate Template. In the Duplicate Template dialog box, select the Windows Server 2003, Enterprise Edition template version.
      3. Click OK. The Properties dialog box for the certificate template opens.
      4. On the General tab, in Display Name, type a name like ‘AMT 802.1X Client Authentication’.
      5. Click the Subject Name tab, and then click Supply in the request. Click OK in the warning dialog box for this setting.
      6. Click the Security tab, and remove the Enroll permission from the security groups Domain Admins and Enterprise Admins.
      7. Click Add, and add SCCM Primary Site Servers.
      8. Select the following Allow permissions for this group: Read and Enroll.
      9. Close the Certificate Templates Console.
      10. Force replication across Domain Controllers using Active Directory Sites and Services. Begin replication from the server noted earlier.
      11. In the Certification Authority Console, right-click the Certificate Templates node. On the Action menu, point to New, and then click Certificate Template to Issue.
      12. Click AMT client template, and then click OK.
  2. NPS server
    1. Ensure that the NPS server is a member of the ‘RAS and IAS Server’ group in Active Directory.
    2. Restart the server so it can autoenroll the newly assigned server certificate. If the certificate does not autoenroll, open a Certificates MMC session on the server and manually enroll the machine.
    3. Open the Server Manager Console and expand the tree to Roles | Network Policy and Access Services | NPS (local) | Policies | Connection Request Policies. Create a new Connection Request Policy. Name the new policy ‘Secure (Wired) Ethernet Connections’. Leave the type of network access server as Unspecified. Click Next.
    4. Add a new condition: NAS Port Type. In the popup, under Common 802.1X connection tunnel types, select Ethernet. Click OK, then Next.
    5. On the Specifiy Connection Request Forwarding, ensure ‘Authenticate requests on this server’ is ticked and click Next.
    6. Continue clicking Next until the Finish option appears, then click Finish.
    7. Verify the new policy is enabled and listed as Processing Order 1.
    8. In the tree, select ‘Network Policies’ and create a new network policy name ‘Secure Wired (Ethernet) Connections’.
    9. Type of network access server is ‘Unspecified’.
    10. Under Conditions, set ‘NAS Port Type’ to ‘Ethernet’, and ‘Windows Groups’ to ‘Domain Computers’ and ‘Domain Users’. Note that the ‘Windows Groups’ criteria should be a logical OR.
    11. Access permission is ‘Granted’.
    12. Add EAP Type ‘Microsoft: Protected EAP (PEAP). Verify the settings have a certificate association, ‘Enable Fast Reconnect’, and EAP subtype of ‘Secured password (EAP-MSCHAP v2).
    13. Leave the default ‘Less secure authentication methods’ as is.
    14. Do not configure any constraints
    15. Add the following Standard attributes (if you want to pass VLAN settings via the NPS server):
      1. Tunnel-Medium-Type: 802
      2. Tunnel-Preference: 1
      3. Tunnel-Pvt-Group-ID: <ID number or name of VLAN>
      4. Tunnel-Type: VLAN
    16. Set the newly created policy at Processing Order 1.
  3. SCCM configuration (If using Intel AMT, otherwise go to next section)
    1. In the Configuration Manager console, navigate to System CenterConfiguration Manager / Site Database / Site Management / <site code> – <site name> / Site Settings / Component Configuration.
    2. Right-click Out of Band Management, click Properties, and then click the 802.1X and Wireless tab.
    3. To configure 802.1X authentication for wired networks, select Enable 802.1X authentication for wired network access, and then click Set.
    4. In the Trusted Root Certificate for RADIUS Authentication dialog box, specify the trusted root certificate by selecting an enterprise CA from the forest. Ensure that From certification authority (CA) is selected, and select the CA from the drop-down list.
    5. Use the drop-down box to select PEAPv0/EAP-MSCHAPv2 as the client authentication method.
    6. Click Use client certificate to use a client certificate for authentication.
    7. Click Select, specify the Issuing CA to use for the client certificate and the RADIUS client certificate template ‘AMT 802.1X Client Authentication’, and then click OK.
    8. On ‘Out of Band Management Properties’ window, click OK.
  4. Group Policy
    1. Create a new group policy
      1. Computer Configuration | Policies | Windows Settings | Security Settings | System Services. Define ‘Wired AutoConfig’ to Automatic with default security.
      2. Computer Configuration | Policies | Windows Settings | Security Settings | Wired Network (IEEE 802.3) Policies. Create a new wired network policy.
        1. Name the policy
        2. Ensure ‘Use Windows Wired Auto Config service for clients’ is checked.
        3. On the Security tab:
          1. ‘Enable use of IEEE 802.1X authentication for network access’ is checked.
          2. Network authentication method is ‘Microsoft: Protected EAP (PEAP)’.
          3. Click ‘Properties’:
            1. Uncheck ‘Validate server certificate’
            2. Authentication method is ‘Secured Password (EAP-MSCHAP v2)’.
            3. Click ‘Configure…’. Ensure Windows logon option is checked.
            4. ‘Enable Fast Reconnect’ is checked.
            5. Close ‘Protected EAP Properties’ window.
          4. Authentication Mode is ‘User re-authentication’.
          5. ‘Cache user information for subsequent connection to this network’ is checked.
          6. Click ‘Advanced…’:
            1. ‘Enable Single Sign On for the network’ is checked.
            2. ‘Allow additional dialogs to be displayed during Single Sign On’ is unchecked.
            3. Close ‘Advanced security settings’ window.
        4. Close policy properties window.
  5. Network Devices
    NOTE: This configuration assumes that RADIUS is already configured for on the switch for other uses (i.e. SSH logins). If not, you’ll need to create an association in the NPS server and input the server address and shared secret on the device configuration. Chuck Murison has a great blog on how to do this at:
    1. Enable RADIUS for 802.1X (at the global configuration prompt)
      1. aaa authentication dot1x default group radius
        This sets the device to use RADIUS for 802.1X authentication
      2. aaa authorization network default group radius
        This sets the device to use RADIUS to authorize users for specific access
      3. authentication mac-move permit
        This allows machines to be moved while a session is open (device closes old session)
      4. dot1x system-auth-control
        Globally turns on 802.1X authentication
    2. On a PER PORT basis, issue the following commands (a range command may be used to configure ports simultaneously):
      1. mab
        This allows for MAC authentication bypass (where required)
      2. authentication order dot1x mab
        This tells the network device to use 802.1X before MAB
      3. authentication priority dot1x mab
        This tells the network device to prioritize 802.1X before MAB
      4. dot1x pae authenticator
        This prevents the downline client from trying to be a supplicant
      5. authentication port-control auto
        This turns on 802.1X for the port.
      6. Port may need to be shut/no shut to force the client to authenticate.
    3. Follow-on actions
      1. Set the default VLAN for the port to something else (i.e. a guest or offline vlan). The VLAN is automatically switched to the operations VLAN once authenticated.
      2. Set a guest vlan for machines that are being imaged and not yet capable of 802.1X authentication. On the interface:
        dot1x guest-vlan <vlan-id>
      3. Device settings may need to be adjusted to prevent DHCP timeout prior to 802.1X timeout. The following settings were successfully used in the test lab:
        1. dot1x timeout quiet-period 3
          This setting is the idle time between failed authentication and the next attempt.
        2. dot1x timeout tx-period 5
          This setting is the idle time between transmissions.

McAfee Agent Status Monitor on x64/Server Core Systems

The McAfee Agent Status Monitor is normally launched from the system tray.  But if you have a 64-bit OS and/or Server Core installed, there is no icon.

Fret not!  You can still pull up the Agent Status Monitor from the command line:

c:\Program Files\McAfee\Common Framework\cmdagent.exe /s

Running SCCM 2007 Software Updates on Server Core

Updated 2/8/2016

Oh man, this is my best work in months!

Our network has several Windows Server 2008 (and now 2012 R2) Server Core installations.  We also run System Center Configuration manager 2007 to manage software updates.  As anyone who has used SSCM knows, advertised software updates show up as a notification in the system tray, which is fine, unless you’re running Server Core.  That’s because Server Core doesn’t have a system tray!

Now, I don’t know about you, but I’m a control freak when it comes to running software updates on my servers.  I only want them to run one at a time, when the server is offline for maintenance, and with me there watching the updates happen.  So that means a mandatory advertisement to my Server Core machines was out of the question.

I started searching the Interweb for solutions discovered by similarly-situated system administrators.  Unfortunately, this is what I found:

– People (trying to be helpful) answering the question (or posting snarky responses) when they didn’t even understand the question… this makes me absolutely crazy (c’mon folks… either help up or shut up)!
– Powershell scripts invoking other Powershell scripts invoking…
– Some weird thingy using Maintenance Windows to stagger updates across machines

Ugh!  I wanted a simple, elegant solution.  I didn’t want to do any of this.  So after I thought about this for a few days, I realized you could use a task sequence as a proxy to initiate updates (since control panels are still available in Server Core):

1. Create a Software Update deployment like you normally do and assign it to your Server Core machines.  Make sure the deployment is not mandatory.
2. Now, create a Task Sequence.  In it, add a single task: Install Software Updates.  Be sure All Software Updates is selected.
3. Advertise the Task Sequence you just created to your Server Core machines.  Ensure the advertisement has progress display checked.  Check the SCCM log to make sure the advertisement is pushed before continuing.
4. On each Server Core machine you want to update, perform the following:

a.  Open the SCCM control panel: c:\Program Files\SMS_CCM\SMSCFGRC.cpl (on 64-bit OS, this will be in Program Files (x86) [UPDATE: SCCM files on 64-bit machines may be located at c:\WINDOWS\SysWOW64\CCM instead].
b.  On the Actions tab, initiate a Software Updates Scan Cycle.  This can usually take a few minutes.  Check ScanAgent.log for status
c.  On the Actions tab, initiate a Machine Policy Retrieval & Evaluation Cycle.  Wait a few moments for this to complete.
d.  Open the Run Advertised Programs control panel: c:\Program Files\SMS_CCM\SMSRAP.cpl (again the path for 64-bit will differ)
e.  You should see your advertised task sequence to run software updates.  Initiate the task and watch it go!

Ha ha! No code and no weird settings!  And, I get to reuse it over and over again!

P.S. Be aware that there is no way to suppress a forced restart for a task sequence if the update package requires it.  So make sure your server is ready for a reboot when you start your updates!

SCCM 2007 Active Directory System Discovery Agent DDR Errors

If you are getting DDR errors during AD System Discovery, check to see if there are any cluster objects within the tree that the service is discovering.  Since these are not actual hosts, they will flag as having inaccessible properties.  The fix to this is to either exclude that OU from your discovery or better yet, deny access for the CCM server to that OU by way of AD Users and Computers security configuration.  This effectively hides the OU from the CCM server.

SCCM Site Configuration for Windows Server 2008/2008 R2

Was just finishing up creating a secondary site and started getting an error in the Distribution Manager component.  Turns out I forgot to enable RDC, which is disabled by default in Server 2008.  There is a Technet article that explains what you need to do to use 2008 for SCCM.  Don’t forget to do these!

Thanks to Teh Wei King’s System Center Blog for the solution.

Black login box on Windows Server 2003

If you run out of space on your Windows 2003 Server, you might end up with a login box that is black. This is addressed here:

IE 8 Search Provider Default Error (Group Policy Conflict)

Our Group Policy turns off IE search on our Windows 7/Server 2008 RTM/R2 machines.  We noticed that when a user first logs into one of these machines and opens IE, it displays a message that:

A program on your computer has corrupted your default search provider setting for Internet Explorer

The GPO deletes the search provider settings (since they are not used).  IE interprets this as a fault.  Microsoft has a hotfix for the problem:

Broadcom HBAs do not work with HP P4000 SANs

BLUF – If you use LeftHand/HP P4000 SANs, DO NOT purchase the iSOE option on Broadcom NICs, it doesn’t work.  At least not yet.

On our last purchase cycle, we decided to try out Broadcom’s iSCSI Offload Engine (iSOE) option on their NetExtreme II 5709C NICs.  Well, it turns out that their version of iSOE doesn’t play nice with the HP P4000 SANs.  The only HBAs supported are ones made by QLogic.  I asked HP for details on if and when HP plans to support them.  It appears that they are working on supporting the feature, but no timeline on release just yet.

We have successfuly used the TCP Offload Engine in our production environment for quite some time now, so that’s still a good option.

Here is the link to HP’s Compatibility Matrix (I guess I should have looked at that first):