Friday, January 17, 2014

SCCM Updates: Go for "Autopilot" not "Total Automation"

I've been getting lot of questions on a previous post regarding my approach to software update structure with SCCM. I replied to a few and this generated even more questions so I felt the need to do a more lengthy response about why I do things the way I do them and what led to my approach and that led to this post. I'll start by answering the last question posted there. Yes, I clean up superseded and expired updates by hand (for now at least) and there are a few very good reasons why I do it that way. When I was first attempting to implement software updates with SCCM I feel that my big initial mistake was to chase after a strategy that focused on "total automation". During my that first approach I literally created an ADR and package for every individual product that I wanted to patch. The ADRs ran monthly and cleaned out the expired and superseded stuff automatically. Sounds great right? In practice what I actually discovered was this approach works great for something simple like a daily Endpoint Protection update (because in that scenario you really only have to consider about 3-4 small patches a day tops). You try to do that with all the security, critical, service pack, etc. updates for multiple Windows versions going back to say 2003 and you run into several problems.

First you have a *lot* more patches to deal with in your software update group(s). Every time your SCCM clients kick off a software update evaluation cycle they have to look through all those updates and determine if any apply to the device upon which they are running. At a Microsoft conference last year I learned that this can cause high CPU utilization on your target machines. Still that’s not too bad because it only has to run once against the software update group provided the update group doe not change after that initial run. Ah but there’s the rub, I also learned that every time the ADR runs it automatically changes the content in your software update group! Do that every Patch Tuesday and your clients have to once again parse or scan that update group which (again) can hit your CPUs pretty hard. To put it in perspective, think about how many times we hear users complain about "slow computers" in our line of work. You don’t want to introduce something hat will hammer CPU utilization on your Servers or User PC multiple times a month.

Second by having a bunch of ADRs (I had a total of 12) that were constantly changing and updating the content in my software update groups I discovered I was putting a lot of load on my primary SCCM server in the weeks following patch Tuesday. The reason there should be obvious as it was constantly downloading patches and cleaning up software update groups based on those ADRs. So by chasing total automation I was (in fact) causing lot of performance issues on both my clients and my servers. Also it really made no sense to have ADRs that constantly ran against groups with updates that go way back in the past (they almost never change unless Microsoft expires or supersedes them). I came to realize that year-based static update groups with a single monthly ADR were the best choice for dealing with the performance and organizational issues I was facing. The reason being is I typically only have to change the older groups once a quarter (sometimes longer).  So (usually) I only have to worry about the updates in my most recent year-based group or my monthly groups (I usually keep monthly groups up to three months in the past before moving the updates from the oldest group into a year-based group).

Finally I learned another valuable lesson about the ADRs. No matter how well you craft your download rules they will inevitably screw something up. Either they will download something you don't want or remove something you want to keep. Its much better to have your monthly ADRs downloading updates into a test group and then (once you've verified what it downloaded) you can then move the contents of that test group into your static update group that targets production machines. Coincidentally this solves the "Expired and Superseded" issue through attrition as it forces you to do a "monthly visual audit" on your update groups when you open the console up after patch Tuesday to see what your ADR downloaded. If your static groups contain superseded or expired updates you'll see that the icon next to their names has changed to reflect that. At that point its child's play to just open the group and remove the older updates by editing their membership.


So that’s a pretty big reply to the questions I was being asked and (honestly) a lot of it is just what I’ve discovered through using SCCM 2012 in the past year. There may be a better way to do it, but this one works for me. I really struggled a lot with this at first and I hope I'm posting this big response in an effort to help others avoid some of my own pain. SCCM is a really complicated animal and I just think its a bad idea to go into it "set it and forget it approach". If it were really that easy we wouldn't need SCCM admins right? The main point I’d like to get across though is sometimes a 100% automated solution just doesn't make sense. SCCM can automate a lot of things for you but even with the advent of the autopilot feature on an airplane we still need pilots at the helm. Focus on design paradigms that automate the most time consuming, tedious or repetitive steps. Then if you find yourself still stuck with a manual step, try to make it something you can do very quickly (like at the touch of a button). I hope that makes sense and that others find it helpful. If there are any counterpoints to this philosophy or if anyone just has better ways of approaching this I'd love to hear about it in the comments section.

UPDATE: Someone requested a screenshot of my SUGs. I've added it below. Like I said we roll about 2-3 months in the past on our monthly deltas and once we achieve around 90% I move those to a year-based group. I've highlighted the delta and year-based groups in the screenshot. The other groups listed there are "special case" groups:


56 comments:

  1. Thanks ZeusABJ

    I'm trying to get a handle on our SCCM, and it's been more difficult than I imagined.

    I've read several SCCM 2012 books, but I just don't seem to grok the logic...

    1) How do I consolidate old updates into a single (by year) update group?
    2) After that, what is the safe way to delete the old (randomly named) groups?

    ReplyDelete
  2. Just use "Edit Membership" to move the updates between groups. Delete your old groups once they are empty. Be aware that if you use edit membership and move the updates into a group that is associated with a different package you may have to re-download the updates into the package/folder combination for the software update group you have moved them into.

    ReplyDelete
  3. Again Thanks. Great articles, Thanks for sharing man.. It's like I understand how this works now. Can thank you enough.. Owe you another beer..

    If you have the time could you write about you're experience installing software updates during a Build and Capture for windows 7. I'm dealing with it as we speak, and my issue is that 190 updates or downloading and installing, but hangs after the last update is installed, Read a lot about similar issues, but didn't found a solution. yet..
    Trying to do it like this: http://www.toolzz.com/?p=1059
    Thanks

    ReplyDelete
  4. We are not using OSD, only MDT so our Build and Capture process happens outside of SCCM. Sorry no help there.

    ReplyDelete
  5. Great article. One of very few Software Update articles that makes sense and introduces simplicity. My question is regarding the ADR Deployment Package for the monthly update. How often do you remove the old updates from the Deployment Package? As each months updates will append to the previous months updates in that package. Thanks

    ReplyDelete
  6. Typically I remove expired updates right away. Superseded I may leave for a month or two.

    ReplyDelete
  7. I'm newer at SCCM and am having a lot of questions marks come up as I am trying to fine-tune my ADR package. Currently I only grab "Critical" and "Security" Updates for the "Last 1 Month" but I feel like I may be missing updates doing it this way. I know this differs between organizations but could you share your ADR - preferable with some explanations :-) - to give readers some ideas for crafting their own ADR?

    Thank you very much!

    ReplyDelete
  8. Hey Gswipe,

    I basically do what you outlined above only I also do update rollups as well. My advice to you is make sure you check your products under the software updates tab and make sure you have everything you want to support there. Honestly in my (experience) the ADRs will never be 100% accurate all the time. I just look at them as a way to automate Most of what I need. Often I find myself clicking the "All Software Updates" item in the left pane of the console and running a search with expired and superseded set to "no" and then sorting on the date column. I'll then give the list a visual inspection and add any updates the ADR may have missed.

    Beyond that once a year in January I’ll do a “year end review/cleanup” on my update groups (I’m actually doing for 2015 right now) where I create a new year-based update group for the previous year that has gone by. When I do this I do a search for all updates in the previous year (all 12 months) for any products we support. Re-download them manually and put them into productions during our regular monthly update deployment cycle (so January also becomes my “catch up month” for any updated that might have been missed the previous year. I think you’ll find thought that going monthly will work just fine (for the most part). Like any good pilot when you put something on autopilot you should always periodically check to make sure its working as expected from time to time and deal with any “drift” that moves you off course.

    ReplyDelete
  9. I too struggle with grasping the concepts. A consultant had helped us when we first started and we have 1 ADR that downloads updates on Patch Tuesday. This typically will create a few SUGs per month.
    Our current process is we right click on each SUG and deploy it to our production systems (workstations or Servers)
    Any given month there are a total of 3-5 SUGS for each month.

    Now what I am trying to do is move to a more monthly/yearly process, kinda like what your doing.

    So here is what I think you are doing:

    1. Create 1 ADR that is monthly (Patch Tuesday)
    2. That ADR creates SUGs that contain all the updates with the settings you set.
    3. You manually created a SUG that is named yearly
    4. You manually created a SUG that is broken up in 3 month chunks
    5. You take all the new updates from patch tuesday and change their membership to reflect the 3 Month SUG and this gets deployed to your production machines.
    6. After the 3 months you move those updates into the Yearly SUG.

    Does this make sense? Am i on the right path?

    I am thinking of creating an ADR for Servers and one for Workstations, and then have the corresponding SUGS (monthly and yearly) for each OS type for better clarity.
    I find when i have all of them mixed together it can be challenging going through each one to make sure everything is good.

    ReplyDelete
  10. Sort of, here is is in a nutshell:

    1. Create 1 ADR that is monthly (Patch Tuesday)
    2. That ADR creates a single SUG for the month the ADR runs.
    3. The monthly SUG contains all the updates with the settings set in the ADR.
    4. The monthly SUG is auto-deployed to a pilot group the moment it is created.
    5. Pilot machines are checked to verify nothing is broken by updates.
    6. Additional deployments for production machines are created on the verified monthly SUG.
    7. These deployments are created manually to ensure the update schedule is visually approved bu a person (i.e. not fully automatic).
    8. This process is repeated for two more months.
    9. When a SUG reaches 3 months of age we check for at least 85-90% compliance.
    10. If compliance requirements are met the updates are moved into a year-based that is in a constant state of deployment (like an "archive" for in production updates, this means any machines that get on the domain that are non-compliant will be brought up to compliance automatically and if a user removes a patch SCM will put it right back on there the next time it hits its polling interval).

    Thats our strategy. You may have a better one but this works for us.

    ReplyDelete
  11. Thank you for your great blog and replies.

    "Every time your SCCM clients kick off a software update evaluation cycle they have to look through all those updates and determine if any apply to the device upon which they are running. At a Microsoft conference last year I learned that this can cause high CPU utilization on your target machines."

    After step 10, it will change the year-based software group. I assume that your production SCCM's clients is your target of the year-based software group deployment.

    Do you face high CPU utilization on the production SCCM's clients every 3 months after step 10?

    ReplyDelete
  12. Not normally no. I hear it is possible though if you have more than 1000 updates in your update group. My team tracks this for me and keeps them under 1000 at all times. If we have to go over 1000 updates the plan is to split the group but (so far) we haven't had to do this.

    ReplyDelete
  13. Hi, I'm responsible for setting up Windows Updates for a hospital with 5000+ active clients. My idea was to create collections for each department in the hospital so we had more control of when updates were pushed out based on maintenance windows, etc..

    I really like your current strategy and I'm wondering if you'd see any issues with adopting it for our environment with that many collections/clients?

    ReplyDelete
  14. This comment has been removed by the author.

    ReplyDelete
  15. Hey Andrew,

    Wow 5000+ clients, thats pretty awesome! I'd love to have a set that large under my belt. Max I've done thus-far is about 2400 (not chicken feed but still less than half of what you've got). Anyway, thanks for the kind words, the best advice I can give you for a install base that large would be to consider multiple DPs for load balancing. I don't know how may of my other posts you've read but I recently learned at an Ignite conference that some of the "experts" are now recommending approaches very similar to the one I outlined here. I think this approach is just a good overall "best practice" and I'm still using it myself today.

    The only thing I'm doing differently is trying to find ways to use PowerShell to automate the manual step of creating the multiple deployments off the SUG that my ADR creates but even in that case I'm planning to run the script manually. I just think its a very good idea to put eyes on what you deploy first, rather than trust an automated process entirely. Remember you are the pilot, and autopilot is not infallible. It sup to you to make sure that plane does not crash. If you try it out and it works for you drop me another comment here. I'd like to know I helped contribute something to managing to a 5000+ client install base.

    ;)

    Good luck!

    ReplyDelete
  16. Thanks for the info. I realized after reading this post that I was applying ADR's wrong. What I was doing was right-clicking on the ADR itself and choosing Add Deployment. Then I would just pick another collection. Apparently that's the wrong way to do it!

    ReplyDelete
  17. Ah gotcha. If you are having trouble getting started Kent Agerlund has some great books to help:

    https://www.amazon.com/Kent-Agerlund/e/B008A01KS8

    I've attended a number of conferences where he spoke and he really helped me grasp the fundamentals of SCCM. Learning to think the "SCCM way" can be a bit challenging at first.

    ReplyDelete
  18. Do you set it up for all updates to go in one Deployment Package? Or are you creating a new deployment package every month?

    ReplyDelete
  19. Currently I have a new SUG created every month. I am looking at switching the old ones to go in yearly SUGs. Let me know if I'm doing this wrong:

    1. I view the members of an old SUG
    2. Highlight all updates
    3. Edit membership

    When I do this there are many older SUGs that are these updates belong to (they are checked when editing membership). At this point do I uncheck them all and then check the new yearly SUG?

    ReplyDelete
  20. Hi Tyler,

    Question #1:
    No I do not put all my updates in one package as that would cause performance issues. In most cases create a package for each SUG, but (occasionally) I do wind up with a few related SUGS that may only contain a few updates. When that happens I pay just put those in a single package. A great example of this is Adobe Flash and Reader updates. I put those in a single package called "Adobe Updates".

    Question #2:
    Yes, when you have updates that span multiple SUGs you edit membership uncheck any SUGs you don't want them to be in and only check the SUG that you do want them to be in. Be aware that if the SUG you move the updates to is associated with a separate package then you may se a red X by the updates after you move them. Don't panic, just highlight any update with an X and re-download them into he new package and you should be good to go. Also don't forget to clean up the UNC path to your old package file as you'll have some orphaned update files left over from the move.

    ReplyDelete
  21. Thanks ZeusABJ,

    I'm stumbling also with Updates, maybe any advice. I've created (based on readings) 4 SUG (>2013, 2013-2014, 2015 and 2016)

    I can create multiple deployments to different Collections with different installation deadlines)

    but do I need to deploy all updates also to the all systems collection as available? or to the all unknown computers collection, because all new Computers added to SCCM also need those updates (maybe when installing or when a new collection is created) I'm on the move from a manual environment to an automated installation adn reboot requirement.

    So I can manually create the deployments, no problem, but need a way to get and keep all machines up to date, including newly added machines.

    if anything is not clear please respond and I try to clarify

    Thanks in advance

    ReplyDelete
  22. Hi mate,

    Thanks for sharing your approach wit this, it is very similar to ours. One question I have for you is how you deal with picking up updates that may be missing from your baseline? For a simple example lets say that an admin installs a particular version of .NET on a server that has not been installed in your environment before. Updates for this version would have never been picked up before and placed in your baseline and will not be picked up in the future due to you limiting your monthly update ADR to Updates released/revised in the last month.

    ReplyDelete
  23. Hi Steve,

    I have a saved search that I run once per quarter with the criteria "Expired = No", "Superseded = No" and (this is the important bit) "Required is greater than or equal to 1". This search should return any updates that are "required" by a system in my environment that I may have missed. I then add them manually or tweak my ADR to account for them next time. Hope that helps.

    ReplyDelete
  24. could you send me a screen shot of a setup for software updates as explained to bfsteel@hotmail.com so I can get an idea of what you have explained. New to this software now. Thanks

    ReplyDelete
    Replies
    1. Hi BFSteel, I have posted a screenshot of our SUGs above. Hope that helps.

      Delete
  25. Hi,
    I really like this approach. I've had to read it a couple of times to get the gist of what you were trying to achieve and it does make sense by limiting the amount of client queries etc
    I'm still not 100% clear on how the yearly SUG's are applied after the monthly SUG's are moved into the yearly SUG's.
    I understand the monthly SUG's deployment etc and the once every 3 months moving all updates into the yearly SUG if compliance is deemed within your tolerances.
    The part I don't understand is "the yearly SUG's are in a constant state of deployment". Could you expand on that? ie what schedule are they set on?
    I like the concept that the current yearly SUG is only updated 4 times per year (once every 3 months) which thereby limits the client traffic to those 4 instances and most clients will have all those updates anyway. The yearly SUG is also a great catch all too with the previous years not changing at all, thereby limiting the traffic further.
    So, just curious how the yearly SUG's are scheduled or deployed.
    BTW, great post and we are going to implement this strategy and thank you for saving us this potential pain as we were heading down that same initial road.

    ReplyDelete
  26. Hi louis,

    I think you may be overthinking this. At its core this guide is just giving you a method for "organizing" updates you have already deployed. This "system" was my way of working around basic SUG limitations. The "Year" based SUGs really sort of serve as an "active" archive of updates you have previously deployed. When one of your delta SUGs reaches around 90% compliance you move those updates to a year based group and just deploy it. As the year progresses keep doing this as your subsequent deltas also reach 90%. Since the updates are already on 90% of your computers it does zero harm to leave them "constantly deployed" as they won't reinstall on computers where they are already installed.

    Doing this also lets you clean up all those monthly SUGs that get auto-generated by your ADRs and allows you to consolidate them by year. Leaving them constantly deployed enforces compliancy in your environment because if a user removes an update you previously deployed then SCCM will just reinstall it for you the next time the client runs its Software Update actions. You don't really have to organize them by year or month at all, I've just found that (in practice) going based on increments of time works best for me to keep our SUGs simple, organized and under the "1000 updates per SUG" limit recommended by Microsoft. If that limit were not a factor then you could just ditch the year-based structure, create one big SUG called "Updates we already deployed that we don't want people to uninstall" and deploy that with no end date at all to get the same result.

    In fact, something like that may be a possibility sooner rather than later. Microsoft's decision to switch to monthly cumulative update rollups (i.e. pseudo-service pack updates that contain the latest patch Tuesday updates with all previous updates rolled in as well) may actually make my year-based model obsolete 9as again the main goal was to create a system whereby my SUGs always had less than 1000 updates in them). Also you should make it a point to clean out old updates from the year-based groups as Microsoft expires or supersedes them. I actually had a case last year where three of my really old year-based SUGs wound up with less than 200 updates in each ot them due to expiration ans supersedence! So I took the updates in those three update groups and consolidated them into a new year-base group named 2010-2013 Updates and redeployed it.

    Again this is just a model for structuring updates you already deployed to ensure they stay deployed without blowing up the CPUs on your clients. I hope that makes sense.

    ReplyDelete
    Replies
    1. Hi, thanks for the quick reply. I was thinking about it again and it did make sense. My issue was that we hadn't deployed any updates at all with SCCM as we're only just putting it in. I've now created the groups and am in the process of deploying the SUG's.

      Delete
  27. Awesome Share , liked your approach , Kudos ! . If possible can you share whats criteria you set to create monthly update groups ?
    tx

    ReplyDelete
    Replies
    1. Hi Raheel,

      I feel like I already did that... can you be a little more specific as to what you want to see?

      Delete
    2. Hi, little confuse with setting up ADR for monthly patch tuesday.

      Is it like we create manually yearly SUG (All updates 2017), then we create ADR for monthly , that creates auto group ? What about the package created yearly how we keep that just one , trying to understand your way.

      How do we archive these then

      Delete
  28. Create an ADR to download and deploy monthly "patch Tuesday" updates. The ADR will generate your Monthly SUGs with the updates you have chosen. As the updates in the SUGs get to 80-90% compliance "age" them out of production by moving them to a manually created Year-based group. The year-based group should stay deployed all of the time (so your approved updates will always reinstall, even if someone tries to uninstall them - thats the "archive" bit). Occasionally Microsoft may expire or supersede updates in your year-based groups. When that happens, clean them out.

    Can't make it any simpler than that.

    ReplyDelete
  29. Hi,

    Many thanks for your blog. I'm in the process of switching from WSUS to SCCM, and it's certainly an interesting learning curve. Your blog has given me plenty of food for thought :)

    A couple of quick queries for you. In your screenshot of SUG's, I don't see a separation of Servers VS Workstations. Do you deploy both server and workstation updates via your Monthly Delta updates, and would there be a reason you wouldn't split them (other than trying to keep SUG's to a minimum)? Similar query also for Pilot/Test VS Production deployment - how do you deal with that?

    Many Thanks, Steve

    ReplyDelete
  30. Yes I keep all my updates (regardless of product type) in time-based SUGs. So all updates for Office, Windows, CRM, Visual Studio, etc. are all contained in the same SUG for the year (or delta month) in which they were released. I have some degrees of separation using different collections and deployment types (off the SUGs) but a Windows system running an SCCM Client is capable of sorting out and installing only the updates it needs from those contained a SUG.

    ReplyDelete
  31. Hi Zeus,

    Great blog....I've been bashing my head over SCCM and you given me some great insights :) I have a couple of queries I'm hoping you might be able to shed some light on:

    When creating ADR's and selecting the Product, how do you deal with the multiple products that might be on one device? As an example, I have devices with Windows 10 and Office 2010. If I add those to my Product selection it will show me the updates for those. But then it will not list things like Silverlight, MS_SQL and anything else not included in the product selection that may be installed. I know you said you don't do ADR's for all the different products, so what is your solution? I know I can just adjust the product selection to add the others, but I know something will get missed if a new piece of Microsoft software is installed on a device that I'm not aware of, so my ADR won't catch it.

    My other query relates to ADR's and how to pick up older updates. When you choose "Date Released or Revised", you can only select up to "Last 9 Months" as the maximum. I've been using MS's Baseline Analyzer to see what I'm missing and I can see a few updates (pretty much only Office 2010) which are much older than 9 months. Do you have any thoughts on how to grab those?

    The ADR search criteria seems to be fairly restricted - it would be good if it had the ability to build your own query, like you can in Device Collections.

    Many Thanks,
    Steve

    ReplyDelete
  32. Hi SMJ,

    I put all my products (Office, Silverlight, Windows, CRM, etc.) in 1 big "Patch Tuesday" ADR that downloads the updates, creates my SUGS, and creates my deployments to collections (but they are disabled). After the ADR runs I visually review the content of the SUG, remove any updates I don't want and add any updates the ADR may have missed (they are seldom perfect). Once satisfied with my update content I check the schedule for each auto-generated deployment under the SUG and tweak that (if needed). I then enable each deployment, kick back and watch them go. Once they reach 90% I move them to a year-based SUG that is always deployed. If a user uninstalls an update or if they install something new that needs patching the update just goes out when the SCM client runs a software updates evaluation cycle.

    For older updates I just created some custom search criteria that I run against all software updates a couple of times a month (mostly based on the "is required" option. If I see something got missed or some newly installed Microsoft product in the environment now requires patching I just download them manually and move them into the delta SUG for that month and they go our with the regular Patch Tuesday updates.

    Like I said, this is not full-auto, you are a pilot directing an automated system. Occasionally you'll have to take manual steps to "right the course" of the ship.

    ReplyDelete
  33. Hi Zeus,

    Thanks for that - Awesome, gets me a lot closer to where I need to be. I wasn't quite able to get my head around those last two points. SO now I'll end up with something like:

    1) Create ADR's which includes ALL known OS's and Products (Server & Desktop) to be patched.
    2) Run regular manual searches (with Expired=No, Superseded=No, Required >=1) to pick up missing updates and any newly installed products.

    And maybe a step 3 of going back and re-tweaking the ADR to include any new Products found in step 2.

    I like many on here owe you a beer (or maybe in your case a great cuppa' tea) :)

    Steve

    ReplyDelete
    Replies
    1. Yeah you got it in a nutshell, just keep your SUGs under 1000 updates (if possible). I also have a few extra tweaks to exclude some crap I know I'll never need (like itanium platform updates) Sometimes it takes a little trial and error but just keep tweaking and refining over time and You'll fall into a pattern that works for you. Also your search pattern is a decent start, but (honestly) the ones I seem to use the most are like so:

      - Updates released in the last day with Expired=No, Superseded=No, Title not Definition

      - Updates released in the last week with Expired=No, Superseded=No, Title not Definition

      - Updates released in the last month with Expired=No, Superseded=No, Title not Definition

      Then I sort on "Deployed=No" and just look at anything I don't have deployed. Those usually paint a very good picture of anything I may have missed.

      Ha, yeah I love my tea ... and beer ... and cocktails ... and I really need to update this blog. Life is busy when you are an SCCM admin! I'm currently in the thick of some major PowerShell scripting. Maybe when thats done I'll post some code...

      Delete
  34. Hi Zeus,

    I've managed to get to a point where I'm almost happy, thanks to your blog and additional tips :)

    In your screenshot of your SUG's, your monthly ones are for example - "Monthly Delta Updates December 2016".

    My ADR is called "Monthly Windows Updates", and when it automatically creates the SUG for my pilot users, the SUG ends up with a name like "Monthly Windows Updates 2017-08-11 16:02:51".

    I'm thinking that you are renaming the SUG to what you want it be after it is automatically created - is that correct? If I manually create a SUG, no problem's - but if it is automatically created then I have the date and time appended.

    Many Thanks,
    Steve

    ReplyDelete
  35. Hi Steve,

    I do rename the SUGs (once I'm satisfied with their content) but you don't have to. I also rename the deployments under he SUG as well. Its just my personal preference and it makes them easier to find when searching for them under the monitoring node in the console. It also makes them easier to distinguish when running SQL reports against them. Again though, the renaming is not required.

    ReplyDelete
  36. I've adopted this approach and find it works well.
    I have 3 ADRS, each about 4 days apart for test workstations/servers, then servers, then workstations. The test workstations servers are deployed to a week after the 3rd week of the month when the security quality roll ups are released. We don't bother with just the security only updates. Each ADR is set to a single SUG rather than a new SUG every time it runs as I'm told the ADR will automatically remove expired, superseeded etc when it runs and each ADR holds 3 months worth of updates.
    At the end of the year, I then create a yearly SUG that contains all of the years updates as recommended here, ensuring that they don't ever contain more than 1000 updates.
    Ocasionally, I have to go in and remove the expired, superseeded from the yearly SUGs. Works well.

    ReplyDelete
  37. Awesome Louis! Glad this method works well for you!

    ReplyDelete
  38. Hi Zeus,

    I have most of my SCCM patching now working the way I want, thanks to your help :) I do have one new issue:

    When my ADR ran and created my October SUG, it actually had September patches in it. I have a sneaking suspicion that it's to do with the Deployment Package part of the ADR.

    So my question is to do with the Deployment Package tab within the ADR. The two options are "Select a deployment package" or "Create a deployment package". If you select Create then you need to provide a Package source. I went back and had a look at the properties of my ADR and noticed that it was set to Select a deployment package and it was my September one - guessing this is related to my issue. I don't see a way to create a deployment package manually, so I'm guessing it can only be done via the ADR.

    So how do you configure this part of it? Do you "Select" or "Create" so that it works correctly for the automated monthly process, and do all updates go to the same deployment package each month?

    I've read quite a few documents regarding this, and they usually say you can do it this way, or that - but none seem to go into the detail of how to get this bit automating properly for the delivery of patches to my pilot group.

    Apart from the above issue, everything else is looking great.

    Many Thanks,
    Steve

    ReplyDelete
  39. Point #1:
    It probably has less to do with your ADR than you think. Microsoft has started re-releasing some patches after their "patch Tuesday" debut, also they seem to have started releasing updates for Office later in the month (no idea why there). Plus, sometimes if a month runs 28 days instead of 31 or if Patch Tuesday falls really early in the month your ADR may create a SUG that contains updates that were released in the Patch Tuesday cycle for the prior month. Point is, I've seen this behavior too and it doesn't really matter IMO. Who cares if you redeploy September updates in your October SUG, you already deployed the previous month so it’s not like they will reinstall right? Also, the SUG just contains pointers to the same updates you downloaded in September and they should be contained in the same package (if you followed my guide) so it’s not like you are downloading duplicates. So that part is a non-issue.

    For me the only thing I do watch when that happens is any updates that Microsoft posts AFTER one patch Tuesday cycle but BEFORE the next cycle hits. Those are usually re-issues of previous updates but (sometimes) they are "preview" updates for the next cycle. You certainly don't want to deploy those to your production servers and clients so watch out for those. Who knows what sort of effect they could have on your environment. Also, that just serves to back up my "Autopilot" instead of "Automate" assertion. *Always* review those SUGs before you enable those update deployments. Never trust an ADR to be so well crafted it becomes “set it and forget it”.

    Point #2:
    Yes, the “Packages/SUGS/Download Location” relationship be very confusing at first. The simplest explanation is that a package is simply a container in SCCM that lets you organize the updates you download. The SUGs can then be applied as “filters” or “masks” for the contents of these packages. The SUGs themselves don’t contain the update files, they just reference or “point” to them in the package. The packages then “obtain” or “pull in” the update files form your download location (usually a network share). So, packages are how you store and organize the update *files* in SCCM, SUGs are how you group and disseminate those files in SCCM. SO you could have a single package that contained updates for Windows and maybe an update for CRM that was referenced by two SUGS, one named “Standard Windows Client updates” and one named “CRM Windows Client updates”.

    When I set up SCCM from scratch I typically use software update searches to pre-create my packages, then I just “attach” my ADR to that package by choosing that select option. You don’t want to create a new package every time because (historically)having a lot of packages can be processor intensive for your clients. That’s why I advocate creating year-based packages for all prior years and a single delta package for your current year. When next year hits make your current delta year-based (for the year that just passed) and make a new delta for the new year. I know it’s a bit hard to explain but that’s the best way I can describe it. Point is my SUGs change all the time but my packages seldom do because they are just containers of my updates.

    Does that make any sense?

    ReplyDelete
  40. Hi Zeus,

    Thanks for the update. Yes that makes much more sense now :)

    I wasn't sure whether to split them up so I don't have too many updates in the one location, but it sounds like only the SUG's having too many items causes performance issues.

    I had split them up into a couple of different package source locations (a couple for different yearly updates, and then a couple for different months). It sounds like I should combine them all (desktops and servers) into the one package source location.

    I'm wondering if I can create a new package source location for all required updates, copy the updates from the existing ones into the new one, and then go back and modify the Package source locations to point to the new one. Have you tried that, or know if it sound like it should work?

    If it does, then I can go back and remove the old package source locations from disk afterwards.

    Many Thanks,
    Steve

    ReplyDelete
    Replies
    1. Yes you can do that, although I usually just re-download them. Also if you double-click the package in SCCM it will show you a listing of all the updates contained in it. If you right-click and delete an update in that view it should delete the corresponding file(s) on your share as well. Point being SCCM will clean those shares up for you as your ADRs run and remove old updates. Also if you move your package source to a new location and don't move the files you'll see a red "X" appear on the SUGS that reference those updates. Just go in the SUG, right-click and re-download them and the X will disappear. I had a storage guy change some drive letters on me once and all the X's freaked me out, then I realized I could just change my package location, re-download them and all was well.

      Delete
  41. Hey Zeus - it's me again :)

    Just one last item which I've just discovered while having a look at the content of my package source locations.

    While having a look through the folders, I'm seeing packages for languages other than English having been downloaded (I do also have the English ones so that bit is good). I could sort of tell by the name of the CAB files, but to prove it I opened a couple and extracted the XML file - sure enough I've got Thai, Russian and lots of others.

    I'm not sure how that has happened, as I only have one ADR and it is set to English only under the Language selection. I have done some manual deploys, so wondering if that has any relationship.

    Anything you have experienced, or can see in what I've explained that might show a reason for this?

    Looks like I'm going to have to rebuild those package source locations again anyway :-P I could just review and delete the non-english packages - there's only a couple of hundred, but not sure of any other impacts.

    Thx,
    Steve

    ReplyDelete
    Replies
    1. Yes I have witnessed this behavior as well and no I wouldn't mess with them. I think some updates are just multilanguage no matter what you do. Don't go messing with those CAB files, trust WSUS to download the files it needs. The only time I poke around in there is if I want to get a copy of an update file to run manually somewhere and even then I usually just get it from the Microsoft Update Catalog website.

      Delete
  42. Hi Zeus,

    Thanks for that. At least I'm not going crazy with the multiple languages. Seems Office updates are one of the main offenders for that. Bit of a waste of resources when you only use one language - but that's what MS has given us.

    It looks like I will have to download my patches again and build one large deployment package for all updates. I tried moving the updates on a couple and changing the package source. Initially it seemed to work, but subsequent attempts produce the error "The package source has already been used. Specify an alternate source directory". If I get all my updates in one package source location and just have my ADR add to that I should be good.

    I have a couple more quick queries when you have some time:

    1) Now that the Windows 10 has the monthly cumulative update that contains all prior months fixes, when you move these to your yearly SUG do keep each of the prior months versions as well or only the latest/ Just wondering, as each of those is over 1 GB in size and over time that will take a bit of storage, especially when we have the same for server OS's as well.

    2) Are the .Net security updates for Windows 10 included in the monthly cumulative update as well? I've been searching all updates until I'm blue in the face trying to find these as separate updates, but no joy. I can find them as separate updates for all my other OS's (7, 8, Win2K8, Win2K12), but not Windows 10.

    Many Thanks,
    Steve

    ReplyDelete
  43. Yeah, they still release a ton of separate updates for office. Frankly it wouldn't surprise me to see them go the "cumulative update" route with Office as well at some point. From my perspective the CUs just make more sense logistically, but who knows for certain what they'll do right? Not sure why you are having the issues moving the updates between packages (sounds like you are targeting the same package every time by mistake or something) but remember, having one big package for all your updates is ill-advised. The whole reason you are supposed to break them up is because it is CPU intensive for your clients to sift through packages that contain more than 1000 updates (at least historically that has always been true and I have not heard of Microsoft doing anything to "fix" that in SCCM. You don’t want all your Windows Clients and Servers out there getting bogged down with a consistent 100% CPU utilization every time your Software Update Actions run. Again, the relationship between packages, SUGs and download locations cane be a bit confusing at first but I'd like to encourage you to keep experimenting. You'll get the hang of it.

    To your questions:

    1) No, I do not as Microsoft tends to supersede the CU’s for the prior month. Just back in September they actually went so far as to *expire*a Windows CU which (as you should know by now) makes it non-deployable. This actually presented a major issue for us as we typically wait one month out before we push an update to our servers. Because they expired the CU for the prior month I was unable to update my servers the following month (because I had nothing that was deployable for them!). If Microsoft continues this practice going forward we will have to totally rethink how and when we deploy updates to our servers, otherwise they will never update! Also as Microsoft continues to release these CUs they are also superseding and expiring older updates in my Year-based groups. This means the update count in those SUGs is decreasing and I’ve actually been able to consolidate a few of them while still keeping my count below the 1000 mark. So now I am winding up with a set of SUGs that are based on year *ranges* now (like “Years 2012-2014”, or “Years 2014-2016” the point is its always changing and evolving and you have to be ready to adjust with it (again proving my constant assertion of “Autopilot, not Automation”, you have to be ready to “adjust course” as needed).

    2) I’m not 100% sure on this but I do know they release separate .NET-related CUs, and I think these contain all the .NET updates. I just deploy those as well.

    Good luck!

    ReplyDelete
  44. Hi! I've recently started a new job at a local Hospital management company and have been tasked with coming up with a patching solution that will handle 5 hospitals. I am totally new to patching and have been reading as much as I can.

    I read your blogs and have no clue on what you're talking about(expected) but am sure once I have the basics down I can come back and it will all make sense.

    here in this particular blog you've mentioned a book (https://www.amazon.com/Kent-Agerlund/e/B008A01KS8)that will help with the basic understandings. Kent's latest publication is 2014. Do you still recommend this given the dated material or is there something out there better suited for a super "green" guy like myself when it comes to patching. I have a storage/systems/DR background so all of this is new to me.

    Any help is appreciated!

    ReplyDelete
  45. Yes, Kent’s book is still relevant. I totally recommend it. Don’t be intimidated either, it does seem odd at first and my method is one that evolved over time to make sense for me. It was sort of funny to attend a Microsoft conference years later on software updates with SCCM where they described a method very similar to mine as a “best practice”. Point is, if you get in there and start playing around with it I think that (logically) you’ll arrive at a setup pretty similar to mine and at the end it will finally “click” and make sense.

    Good luck!

    ReplyDelete
    Replies
    1. super appreciate the speedy response. Thanks!

      Delete
  46. Sorry to beat this questions down but I'm a little confused on the source for the files and deployment packages and how they relate to your SUG's.

    You mention to not have 1 deployment package for performance reasons. Are you creating a like deployment package named similar to your SUG.

    Example would be, 2003-2010 SUG corresponds to 2003-2010 Deployment Package?

    Also where are is your source folder for the files? Are you storing them all in one folder or do you have a folder that corresponds to the SUG and DP? Example from above would that location be server\sources\2003-2010 Updates? Or do you just have all updates for all products go into server\sources?

    We also release monthly Office updates along with the Windows updates. Should I break up these types of updates? Ex. have all Windows updates 2003-2010, 2011,2012, and then 3 deltas that are current or can I include all Office updates in there as well? Ex All updates 2003-201,2011, ect.?

    Thanks in advance!

    ReplyDelete
    Replies
    1. Question 1: Yes the names I give to my SUGs are very similar to the names I use for packages. Just remember though that (in reality) a SUG is just a grouping (or collection) of updates while packages contain the actual updates you have downloaded. You just use the SUGs as a way to organize those updates for deployment. Also you can have multiple SUGs that reference a single package, so you should name your packages in a somewhat abstract manner. For example, you could have a single package for your monthly delta updates named "Monthly Delta" that is then referenced by three SUGs named "Monthly Delta - Jan 2018", "Monthly Delta - Feb 2018" and ""Monthly Delta - Mar 2018".

      Question 2: My source folders are all contained on a network share named "UPD$" on a file server. I have a folder under there that corresponds to each package and (to keep network paths as short as possible) I give the underlying folders names that are a sort of abbreviated form of my package names with no spaces. So say I have a package named "Year 2017 Updates", the corresponding source folder would be "YR-2017".

      Question 3: Since I base all my update organization methodology on units of time (i.e. when the updates were released). I do not separate them by product. When I first started working with SCCM I tried organizing them by product name and quickly discovered that organizing them based on when they were released just made a lot more sense (for the big picture). That being said I do (occasionally) run into a special case that will lead me to segment an update off into a SUG by itself so I can deploy it to a specific subset of systems . A good example of this is the recently released .NET Framework 4.7. I have a few servers running mission critical apps that will break if .NET 4.7 is installed. So I created a separate SUG named "NET Framework Updates" for it and deploy it in a different manner from all my other updates, but it still references the Year 2018 Updates package. The only real reason to split up your packages is if you wind up with over 1000 updates in a package as (historically) this has caused performance issues with SCCM.

      Delete