Category Archives: SCCM

SCCM (current branch) offline upgrade adventures (1511 to 1606)

Those who have read my blog for some time now, know that our system is a high security, air-gapped network. This can sometimes make administration of the system…. interesting, to say the least. Most frustrating is when applications take Internet connectivity for granted (which unfortunately, is becoming more and more common).

Our latest adventure is working on upgrading System Center Configuration Manager from 2007 R2 (!) to SCCM (current branch). We are in the middle of an install into our test system, where we perform all tasks as if the network has no external connectivity, just like the real thing. Now I have a total love/hate relationship with SCCM… in that I love it when it works, but it’s a total POS when it doesn’t. And right now… I HATE YOU MICROSOFT.

Now, prior to the release of version 1606 as a standalone installer in October 2016, the only way to install SCCM 1606 was to first install 1511 and perform the upgrade within the SCCM console.

We configured the Service Connection Point to operate in offline mode, in order to simulate the real-world environment. Next, one has to wait SEVEN DAYS before any telemetry* is made available. Once you have waited the requisite time, one must then use the Service Connection Tool to export the telemetry and import the updates. Now, this is where the fun really starts….

Fail one: the online documentation for the Service Connection Tool is only for version 1606; there doesn’t appear to be any way to view historical documentation. This is a problem because one of there features described doesn’t even exist in the 1511 version of the tool: the ability to set a proxy connection. WHAT?!? We get stuck trying to connect, only to find out that the only way to get the data was to send one of the engineers home to pull it from their personal network…. after an hour spent wondering why the command wouldn’t take… So now I’m asking if there is actually a company out there, that uses SCCM, and DOESN’T have a proxy? Ya, I couldn’t think of one either. Supposedly, this was fixed in 1606… we’ll see….

Fail two: once we actually had the data, the Service Connection Tool is used to upload the data to the site server, which it did without complaint. The only problem is, the updates never appeared in the Updates and Servicing tab of the console. This was much harder to resolve. Looking in the dmpdownloader.log file, we noticed this:

dmp_log_1

Hmmm…. interesting. Drilling into the directory, both the ConfigMgr.Update.Manifest.cab file and the other CABs where there, just ONE LEVEL DEEPER than where SCCM expected them to be. C’MON GUYS… did you actually test your tools before releasing them? Anyway, moving the CABs up one level to where SCCM expected them to be fixed the issue.

Hopefully, this will help one of the other 0.5% of SCCM users out there who aren’t actually connected to the Internet (and Microsoft doesn’t give a crap about)…. you’re welcome.

* Fail three: we’re also not allowed to pull data down from the air-gapped system and move it to the Internet… ever. This presented a real problem with SCCM, since it now requires telemetry to be sent back in order to receive updates. Microsoft has been less than helpful with this issue, assuring us that the data (some of which is processed via a one-way hash) contains no sensitive information… which I find highly amusing, since there is no way to independently verify the hashed data. We’re supposed to just trust them…. riiiight….. We worked around this by creating a “dummy” site, which is configured in a similar fashion to provide fake telemetry data, which is then used as a surrogate for the real site. Time will tell if this is genius or blows up in our face!

Advertisements

KB2597166 – Microsoft Excel 2010 Security Update FAIL

** UPDATE-3 28 August 2012 ** Our internal tests show KB2598378 does in fact fix the issue.  Be sure to use the correct version (32 or 64 bit) depending on your Office version (NOT Windows).

** UPDATE-2 23 July 2012 ** Microsoft says this issue has been fixed in KB2598378.  We’re testing the fix at our shop now; let’s hope the second time’s the charm…

** UPDATE 17 June 2012 ** It appears that Microsoft has released a hotfix that resolves the issue (KB2598144).  We have NOT tested this hotfix, so I can’t say whether it works or not.  If anyone out there has, please let me know your results…

MS12-030 (KB2597166), was released on 8 May 2012.  This magical piece of crap will cause your Excel 2010 application to BREAK when users try to sort large data sets.  Indications include:

  • Large Operation warning box: “The operations you are about to perform affects a large number of cells and may take a significant amount of time to complete.”
  • Error: “Excel cannot complete the task with available resource.  Choose less data or close other applications.

We were able to recreate by selecting all cells and attempting a sort operation.

Word on the street is that Microsoft is aware of the issue but DOES NOT have a fix.  Their “workaround” is to not select the entire sheet prior to sorting.  My advice: DON’T INSTALL THE UPDATE (unless you like a bunch of angry users chasing you with pitchforks)…

If you happened to have already installed this patch, you can remove it using the following command line (helpful if you use automation):

msiexec /package {90140000-0011-0000-0000-0000000FF1CE} /uninstall {B76D8C6D-1F13-42A7-9931-D7504CB89D6D} /qn

SCCM 2007 Error 2302: SMS Distribution Manager failed to process package

You may encounter this error when trying to update a distribution point.  You may also get errors 2348 (failed to decompress).  This can be due to binary differential replication trying to send a corrupted package.

To fix this issue, disable binary differential replication, update and wait for your DPs to replicate.  This causes the ENTIRE package to redistribute (not just the deltas).  You can then safely turn binary differential replication back on.

Force attempt to provision vPro AMT using SCCM in-band provisioning

If you don’t properly configure your workstation for vPro AMT provisioning before the first SCCM agent call (e.g. you forget to set your certificate thumbprint in MEBx), you’ll end up waiting 24 hours for the machine to reattempt provisioning.  If you’re impatient (like me) you can use this technique to force a reattempt immediately (credit to William York – original source):

Manual Steps to issue WMI command:

  • Open a command prompt and type wbemtestThis is the Windows Management Instrumentation Tester
  • After the Windows Management Instrumentation Tester Utility Opens, click Connect
  • In the Namespace of the Connect Window, type the system name you want to force the check followed by \root\ccmExample: **
  • Click Connect
  • You can also simply run the command on the local system by simply leaving out the host name
  • Example: \root\ccm
  • After you successfully connect to the target system, click the Execute Method Button
  • In the Get Object Path window, type sms_clientin the Object Path fieldClick OK
  • In the Execute Method Window, enter TriggerSchedulein the Method FieldClick the Edit In Parameters Button
  • In the Object editor for _PARAMETERS window, Double Click the sScheduleID in the Properties field
  • In the Property Editor Window, change the Value to Not NULL and add the following {00000000-0000-0000-0000-000000000120}This value is the Object ID to initiate this OOB auto-provisioning check.
  • Click the Save Property button
  • In the Object editor for _Parameters window, click the Save Object button
  • In the Execute Method window, click the Execute Button
  • After you Execute the method, you should see a message that the Method was executed successfully
  • To confirm that your method was executed, look at the target systems c:\windows\system32\CCM\Logs\oobmgt.logYou should now see a new entry in the log GetProvisioningSetting indicating that the policy has been re-evaluated.

Force uninstall of Windows Server Update Services (Server 2008 RTM)

Last week, we had a hardware failure that cause corruption of one of our SCCM site servers.  To clean in up, I needed to uninstall WSUS from the server.  Unfortunately, attempting to remove the role only gave me an error message saying removal failed.  After pieceing together stuff from the web, I think I now have a how-to when it comes to manually remove WSUS.  Since this is the second time I’ve had to do it, I thought it best to write it down this go-around….

  1. First, determine the application’s product code GUID.  You can do this by looking in the registry at \\HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall.  You will need to cycle through the subkeys until you find the one with the correct name in the DisplayName value.  Make note of the subkey; that is the GUID.  To save you the trouble (in case you’re working with the same version as me), the GUID for WSUS 3.0 SP2 is {2C0D7E35-EE6E-4DC7-BA13-2C68AEDEB59D}
  2. Now, at an administrator command prompt, using MSIZAP.exe (which is included in the Windows SDK), clear the install entry:
    MSIZAP T {<GUID>}
  3. Stop the WSUS services:
    net stop wsusservice
    net stop wsuscertserver
    NOTE: If your WSUS install is truly broke, you may just get message saying the services are not online.  In that case, just proceed to the next step.
  4. Delete the WSUS services:
    sc delete wsusservice
    sc delete wsuscertserver
  5. Detele the Windows Internal Database:
    msiexec /x {CEB5780F-1A70-44A9-850F-DE6C4F6AA8FB} callerid=ocsetup.exe
  6. Delete the WSUS program folder: C:\Program Files\Update Service (this may vary if you’re using an x64 OS)
  7. Open IIS manager and delete the WSUS virtual directory
  8. The role will still show in server manager, but can be reinstalled using the WSUS installer package.

SCCM Cluster Site Server error 2147942467: The network name cannot be found

We had recently done an SCCM database site server migration, and this error started popping up:

SMS Site System Status Summarizer still cannot access storage object
"\\server\share$\SMS_server" on site system "\\server". The operating 
system reported error 2147942467: The network name cannot be found.

Upon further investigation,  I realized that I forgot to place NO_SMS_ON_DRIVE.SMS files on our clustered disks (see Microsoft KB 871234.  Because I failed to do that, SCCM installed its components on the disk with the largest amount of available space, which in this case, happened to be a shared database disk.  The error crops up once the clustered disk is moved to another node (since its no longer available on the machine in question).  To fix this error, Microsoft directs you to drop and re-add the site server, after placing the NO_SMS file on the appropriate drives.  But, for a database site server, this is not really an option (unless you happen to have another database available somewhere to make a temporary move).  So, I was now left in a situation where I had to fix it in place.

To do this, first make a copy of the SMS directory to it’s new location.  Then, change the path in the following registry locations:

HKLM\SOFTWARE\Wow6432Node\Microsoft\SMS\Tracing\<Site>\TraceFilename
HKLM\SYSTEM\CurrentControlSet\services\<Site>\ImagePath

On the primary site server, change the path in the following registry location:

HKLM\SOFTWARE\Microsoft\SMS\Components\SMS_SITE_COMPONENT_MANAGER\
Multisite Component Servers\<Target Server>\Installation Directory

Restart the Site Component Manager Service and the issue should resolve.

SCCM Query and Report Aliases

When crafting queries and reports in SCCM (or SMS), it is helpful to know that the WMI class and SQL server naming conventions are slightly different.  This page is most helpful in this respect (of course, it’s hidden deep in the bowels of Technet):

http://technet.microsoft.com/en-us/library/cc180445.aspx

SCCM OOB Management and Intel AMT MEBx (vPro) custom certificate hashes

If you run the Delete Provisioning Data from Management Controller Memory command on a workstation in SCCM, not only does this unprovision the machine for OOB management, but any custom certificate hashes that you entered in MEBx will be deleted as well.  If you want to provision this machine again, you will need to go back in and re-add the hash.

SCCM 2007 SP2 Operating System Deployment and multi-tiered NICs

We’ve begun deploying Windows 7 with SCCM’s Operating System Deployment (OSD) capability.  We’ve found that some of our workstations use multi-tiered LOM NICs.  Turns out Windows PE HATES multi-tiered NIC drivers.  So, if you’re in this situation, you’ll need to get a monolithic driver for your Windows PE boot.  See http://www.windows-noob.com/forums/index.php?/topic/1688-nic-devices-that-require-a-special-driver-for-winpe-may-cause-a-configmgr-task-sequence-to-fail/ for more information.

We’ve also noticed that if you use DHCP, be sure that your leases are long enough to cover the OSD sequence up to the first reboot.  If your lease expires before then, you’ll get cryptic errors (like 0x80072ee7) and your IP address will change to a link-local.  We had a DHCP range that was set to 5 minutes (!) and it was causing this failure… and we had a heck of a time trying to figure out the root cause!

SCCM 2007 SP2 vs. Windows 7 SP1 and Server 2008 R2 SP1

If you’re currently on SCCM 2007 SP2 and have started to deploy Service Pack 1 for Windows 7 and Server 2008 R2 SP1, be sure to grab the MS hotfix that provides support in SCCM for these new versions: http://support.microsoft.com/kb/2489044

EDIT (5/19/2011): Once you install this hotfix, you’ll need to go back and verify any drivers or applications that have OS run restrictions configured.  Because this adds the SP1 operating system as a selection, it will fail to install on SP1 if it is not specifically checked.

Error 0x80092026 during Windows Updates

This error came up when we were trying to send out an update package to our Windows XP machines.  Microsoft has a couple of pages that describe what you can troubleshoot to fix this issue:

http://support.microsoft.com/kb/555374
http://support.microsoft.com/?kbid=822798

None of the suggestions worked for us.  Since our environment is locked-down via draconian Group Policy Objects, I thought that would be a logical place to continue troubleshooting.  After slumming around a bit, I discovered that Computer Configuration | Policies | Windows Settings | Security Settings | Public Key Policies/Certificate Path Validation Settings/Trusted Publishers/Trusted Publishers can be managed by: was set to All administrators only.  This has to be set to All administrators and users for Windows Update (SCCM, SMS) to work properly.

Don’t say I never helped you out…

WinPE SCCM boot image or OSD task sequence failure due to bad NIC driver

Have you experienced a problem loading network drivers into your Windows PE boot image on SCCM, or does your SCCM OSD task fail partway during OS install?  Well, read on!

If you get an error like “Failed to inject a Config Mgr driver into the mounted WIM” or an 0x80070040 “Access Denied” during OSD installation, it is due to the fact that WinPE needs monolithic NIC drivers.  You can verify this if you pull up a command prompt in WinPE (if enabled) by pressing F8 and running an IP config or ping check.  You won’t be able to contact any other machines and/or you’re assigned a link-local address.  For possible fixes, see: http://blogs.technet.com/b/configurationmgr/archive/2010/02/09/nic-devices-that-require-a-special-driver-for-winpe-may-cause-a-configmgr-task-sequence-to-fail-if-a-vista-or-newer-os-is-being-deployed-via-an-operating-system-install-package.aspx

Note that if you load a multi-tiered driver at ANY TIME during the OS install before the machine does its first reboot, that driver will take over and cause the connection to fail.  To work around this, create two driver packages, one without the NIC driver, that is applied during the WinPE phase and another with the NIC driver only, to be applied after reboot.

Running SCCM 2007 Software Updates on Server Core

Updated 2/8/2016

Oh man, this is my best work in months!

Our network has several Windows Server 2008 (and now 2012 R2) Server Core installations.  We also run System Center Configuration manager 2007 to manage software updates.  As anyone who has used SSCM knows, advertised software updates show up as a notification in the system tray, which is fine, unless you’re running Server Core.  That’s because Server Core doesn’t have a system tray!

Now, I don’t know about you, but I’m a control freak when it comes to running software updates on my servers.  I only want them to run one at a time, when the server is offline for maintenance, and with me there watching the updates happen.  So that means a mandatory advertisement to my Server Core machines was out of the question.

I started searching the Interweb for solutions discovered by similarly-situated system administrators.  Unfortunately, this is what I found:

– People (trying to be helpful) answering the question (or posting snarky responses) when they didn’t even understand the question… this makes me absolutely crazy (c’mon folks… either help up or shut up)!
– Powershell scripts invoking other Powershell scripts invoking…
– Some weird thingy using Maintenance Windows to stagger updates across machines

Ugh!  I wanted a simple, elegant solution.  I didn’t want to do any of this.  So after I thought about this for a few days, I realized you could use a task sequence as a proxy to initiate updates (since control panels are still available in Server Core):

1. Create a Software Update deployment like you normally do and assign it to your Server Core machines.  Make sure the deployment is not mandatory.
2. Now, create a Task Sequence.  In it, add a single task: Install Software Updates.  Be sure All Software Updates is selected.
3. Advertise the Task Sequence you just created to your Server Core machines.  Ensure the advertisement has progress display checked.  Check the SCCM log to make sure the advertisement is pushed before continuing.
4. On each Server Core machine you want to update, perform the following:

a.  Open the SCCM control panel: c:\Program Files\SMS_CCM\SMSCFGRC.cpl (on 64-bit OS, this will be in Program Files (x86) [UPDATE: SCCM files on 64-bit machines may be located at c:\WINDOWS\SysWOW64\CCM instead].
b.  On the Actions tab, initiate a Software Updates Scan Cycle.  This can usually take a few minutes.  Check ScanAgent.log for status
c.  On the Actions tab, initiate a Machine Policy Retrieval & Evaluation Cycle.  Wait a few moments for this to complete.
d.  Open the Run Advertised Programs control panel: c:\Program Files\SMS_CCM\SMSRAP.cpl (again the path for 64-bit will differ)
e.  You should see your advertised task sequence to run software updates.  Initiate the task and watch it go!

Ha ha! No code and no weird settings!  And, I get to reuse it over and over again!

P.S. Be aware that there is no way to suppress a forced restart for a task sequence if the update package requires it.  So make sure your server is ready for a reboot when you start your updates!

SCCM 2007 Active Directory System Discovery Agent DDR Errors

If you are getting DDR errors during AD System Discovery, check to see if there are any cluster objects within the tree that the service is discovering.  Since these are not actual hosts, they will flag as having inaccessible properties.  The fix to this is to either exclude that OU from your discovery or better yet, deny access for the CCM server to that OU by way of AD Users and Computers security configuration.  This effectively hides the OU from the CCM server.

SCCM Site Configuration for Windows Server 2008/2008 R2

Was just finishing up creating a secondary site and started getting an error in the Distribution Manager component.  Turns out I forgot to enable RDC, which is disabled by default in Server 2008.  There is a Technet article that explains what you need to do to use 2008 for SCCM.  Don’t forget to do these!

Thanks to Teh Wei King’s System Center Blog for the solution.

Removing a site server from SCCM 2007

System Center Configuration Manager makes you work at removing a site system from the site.  Just wanted to jot down some gotchas that I’ve encountered in the process.

1. Don’t remove the server from the domain before you drop it from your site!  This will cause you to wait at least 24 hours (or do a reg hack) to clear the system.

2. Remove all added roles from the server first – Every role, except component server and site system should be removed before proceeding.  Check the CCM logs to ensure removal is complete.  If you fail to do this, you will never get the option to remove the machine.  The component server role goes away automatically after all added roles are gone.

3. As usual, you have to force SCCM to update.  A right-click refresh on the site systems usually does the trick.  If the component server role still persists, kick the server in the butt by restarting SMS_SITE_COMPONENT_MANAGER via the SCCM Service Manager.  This almost always does the trick.  Some of the older version consoles also required you to exit and restart the console for changes to be displayed, so you can try that too.

4. Assuming everything goes well, the site system will only have the site system role.  At this point a Delete option should appear when you right-click the node.

Good luck…