Transcript
IBM® Security
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide Document version 9.x.11
IBM BigFix Performance Team
© Copyright International Business Machines Corporation 2015, 2016, 2017. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
Contents Contents .............................................................................................................................. iii List of Figures ..................................................................................................................... vii Author List ........................................................................................................................... ix Revision History .................................................................................................................. x 1
Introduction............................................................................................................ 11
2
IBM BigFix 9.x Overview ....................................................................................... 12
3
2.1
Functional Overview.................................................................................. 12
2.2
Architectural Overview .............................................................................. 13
2.3
Return on Investment................................................................................ 14
BigFix Performance Management ........................................................................ 16 3.1
Reference Benchmarks ............................................................................ 16 3.1.1
3.2
3.3
The Unified Benchmark .......................................................................... 16
Key Performance Indicators ..................................................................... 18 3.2.1
Little’s Law ............................................................................................. 19
3.2.2
Evaluating the Number of Concurrent Users .......................................... 19
Monitoring Tools ........................................................................................ 20 3.3.1
Iometer ................................................................................................... 21
3.3.2
BigFixPerf ............................................................................................... 24
3.3.3
BigFixDisk .............................................................................................. 27
iii
4
BigFix Capacity Planning ...................................................................................... 28 4.1
How to Use This Guide: Capacity Planning Guidelines ........................... 28
4.2
CPUs, vCPUs, and Cores ......................................................................... 29
4.3
Capacity Planning Spreadsheet ............................................................... 29
4.4
BigFix Management Server Capacity Planning Recommendations ........ 30
4.5
4.4.1
Message Level Encryption Enablement ................................................. 31
4.4.2
WebUI Enablement ................................................................................ 32
4.4.3
Considerations for a Local or Remote WebUI ........................................ 32
BigFix Console Capacity Planning Recommendations ............................ 33
4.6 BigFix Relay and Associated Infrastructure Capacity Management Considerations ...................................................................................................... 35 4.6.1
4.7
4.8 5
Relay Virtualization................................................................................. 36
Upgrade Capacity Planning Considerations ............................................. 37 4.7.1
IBM BigFix 9.2.5 Upgrade ...................................................................... 37
4.7.2
IBM BigFix 9.5 Upgrade ......................................................................... 37
4.7.3
IBM BigFix 9.5.3 Upgrade ...................................................................... 38
Capacity Planning Example ...................................................................... 38
Performance Management ................................................................................... 40 5.1
5.2
Infrastructure Management Approaches .................................................. 42 5.1.1
Virtualization ........................................................................................... 42
5.1.2
Operating System Management ............................................................. 44
5.1.3
Database System Management ............................................................. 45
BigFix Management Approaches.............................................................. 48 5.2.1
FillDB Options ........................................................................................ 48
5.2.2
Agent Heartbeat Frequency ................................................................... 50
iv
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
5.3 6
5.2.4
Data Archiving ........................................................................................ 50
5.2.5
WebUI Management .............................................................................. 51
Benchmark Management Approaches ..................................................... 52
Database Backup Management ............................................................... 53 6.1.1
Online Backup Support .......................................................................... 53
6.1.2
Database Backup Cleanup..................................................................... 56
6.2
Database Statistics Management ............................................................. 56
6.3
Database Reorganization ......................................................................... 56
6.4
Database Maintenance Automation ......................................................... 57
Security Considerations ........................................................................................ 58 7.1
7.2
8
Console Refresh Frequency ................................................................... 50
BigFix Maintenance Recommendations ............................................................... 53 6.1
7
5.2.3
Security Management ............................................................................... 58 7.1.1
Web Application Security Scanning........................................................ 58
7.1.2
Application Source Code Scanning ........................................................ 59
7.1.3
Threat Modeling ..................................................................................... 59
7.1.4
Security Regulatory Compliance Reports............................................... 60
Security Hardening.................................................................................... 60 7.2.1
Port Management and Firewall Configuration ........................................ 60
7.2.2
Common Vulnerabilities and Exposures Management ........................... 63
Summary Cookbook.............................................................................................. 64 8.1
Base Installation Recommendations ........................................................ 64
8.2
Post Installation Recommendations ......................................................... 65
8.3
High Scale Recommendations ................................................................. 65
v
Appendix A: DB2 Online Backup Enablement ................................................................. 66 References ........................................................................................................................ 69
vi
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
List of Figures Figure 1: Revision History .................................................................................................................x Figure 2: BigFix Architecture.......................................................................................................... 13 Figure 3: BigFix Server Elements .................................................................................................. 14 Figure 4: Business Value Analyst for IBM BigFix and MobileFirst ................................................. 15 Figure 5: BigFix Performance Benchmark Environment Sample ................................................... 18 Figure 6: Little's Law ...................................................................................................................... 19 Figure 7: Monitoring Tools ............................................................................................................. 21 Figure 8: Iometer User Interface Sample ....................................................................................... 21 Figure 9: Iometer Workload Sample .............................................................................................. 22 Figure 10: Iometer Results Sample................................................................................................ 23 Figure 11: Disk Queue Length by IO Subsystem Type .................................................................. 23 Figure 12: BigFixPerf Syntax ......................................................................................................... 24 Figure 13: BigFixPerf Windows Example ....................................................................................... 24 Figure 14: BigFixPerf Windows Example Output ........................................................................... 25 Figure 15: BigFixPerf UNIX Example ............................................................................................. 25 Figure 16: BigFixPerf UNIX Example Output ................................................................................. 26 Figure 17: BigFixDisk Syntax ......................................................................................................... 27 Figure 18: BigFixDisk Windows Example ...................................................................................... 27 Figure 19: BigFix Management Server Capacity Planning Requirements ..................................... 30 Figure 20: Storage Requirements Breakdown ............................................................................... 31 Figure 21: BigFix Message Level Encryption Capacity Planning Requirements ............................ 31 Figure 22: BigFix WebUI v1 Capacity Planning Requirements ...................................................... 32 Figure 24: Console Workstation Capacity Planning Requirements ................................................ 33 Figure 25: Terminal Server Capacity Planning Requirements ....................................................... 34 Figure 26: BigFix Relay Infrastructure............................................................................................ 35 Figure 27: Top Level Relay Capacity Planning Requirements ....................................................... 36 Figure 28: BigFix Virtualization Performance ................................................................................. 37 Figure 29: Sample Capacity Planning Profile ................................................................................. 39 Figure 30: BigFix Capacity Planning & Performance Management ............................................... 41 Figure 31: Modifying the Linux IO Scheduler ................................................................................. 43 Figure 32: Linux IO Scheduler Throughput .................................................................................... 44 Figure 33: Linux IO Scheduler Latency .......................................................................................... 44
vii
Figure 34: BigFix Schema Characteristics ..................................................................................... 47 Figure 35: DB2 Configuration Recommendations .......................................................................... 47 Figure 36: FillDB Database Boost Levels ...................................................................................... 48 Figure 37: FillDB Parallelism Example ........................................................................................... 50 Figure 38: Database Maintenance Approaches ............................................................................. 53 Figure 39: Database Backup with Compression Command ........................................................... 53 Figure 40: Database Offline Backup Restore ................................................................................. 53 Figure 41: Database Online Backup Schedule .............................................................................. 54 Figure 42: Database Incremental Backup Enablement .................................................................. 54 Figure 43: Database Online Backup Manual Restore .................................................................... 54 Figure 44: Database Online Backup Automatic Restore ................................................................ 55 Figure 45: Database Log Archiving to Disk .................................................................................... 55 Figure 46: Database Log Archiving to TSM ................................................................................... 55 Figure 47: Database Roll Forward Recovery: Sample A................................................................ 55 Figure 48: Database Roll Forward Recovery: Sample B................................................................ 55 Figure 49: Database Backup Cleanup Command .......................................................................... 56 Figure 50: Database Backup Automatic Cleanup Configuration .................................................... 56 Figure 51: Database Statistics Collection Command ..................................................................... 56 Figure 52: Database Statistics Collection Table Iterator ................................................................ 56 Figure 53: Database Reorganization Commands .......................................................................... 57 Figure 54: Database Reorganization Table Iterator ....................................................................... 57 Figure 55: Sample Database Maintenance Schedule .................................................................... 57 Figure 56: BigFix Security Management Summary ........................................................................ 58 Figure 57: Security Compliance Report Options ............................................................................ 60 Figure 58: BigFix Port Management .............................................................................................. 62 Figure 59: Port Utility Hosts Configuration ..................................................................................... 63 Figure 60: Port Utility Active Port Configuration ............................................................................. 63 Figure 61: Port Utility Ports and Programs to Ignore ..................................................................... 63 Figure 62: Base Installation Recommendations ............................................................................. 64 Figure 63: Post Installation Recommendations .............................................................................. 65 Figure 64: High Scale Recommendations...................................................................................... 65 Figure 65: BigFix Database LOB Logging Check .......................................................................... 66 Figure 66: Sample Database Backup with Compression Command ............................................. 67 Figure 67: Sample Database Connect ........................................................................................... 67 Figure 68: Sample Migration .......................................................................................................... 67 Figure 69: Sample Database Offline Backup Restore.................................................................... 67
viii
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
Author List This paper is the team effort of a number of security and performance specialists comprising the IBM BigFix performance team. Additional recognition goes out to the entire IBM BigFix development team.
Mark Leitch (primary contact for this paper) IBM Toronto Laboratory
Bernardo Pastorelli Federico Pezzotti Mariella Corbacio Nicola Milanese Pietro Marella IBM Rome Laboratory
ix
Revision History Date
Version
Revised By
Draft
MDL
Initial version for review.
November 1st, 2015
9.2.x.0
MDL
Initial version for publication.
December 10th, 2015
9.2.x.1
MDL
Incorporated review comments (minor edits).
11th,
9.2.x.2
MDL
Incorporated review comments (minor edits).
9.x.3
MDL
Changes made for BigFix 9.5.
April 13th, 2016
9.x.4
MDL
Qualified DB2 online backup support.
May 11th, 2016
9.x.5
MDL
Incorporated updates for the 9.5.2 release.
16th,
9.x.6
MDL
Added storage and anti-collocation recommendations.
9.x.7
MDL
Refined storage and virtualization content.
July 19th, 2016
9.x.8
MDL
Incorporated review comments.
July 26th, 2016
9.x.9
MDL
Added DB2 online backup enablement content.
October 7th, 2016
9.x.10
MDL
Added BigFix 9.5.3 agent resources content.
9.x.11
MDL
BigFix 9.5.5 updates.
July 15th, 2015
January March
June July
July
27th,
6th,
2016
2016
2016
2016
30th,
2017
Comments
Figure 1: Revision History
x
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
1
Introduction Capacity planning involves the specification of the various components of an installation to meet customer requirements, often with growth or timeline considerations. IBM BigFix offers endpoint lifecycle and security management for large scale, distributed deployments of servers, desktops, laptops, and mobile devices across physical and virtual environments. This document will provide an overview of capacity planning for the IBM BigFix Version 9.x solution. In addition, it will offer management best practices to achieve a well performing installation that demonstrates service stability. This will include the deployment of the BigFix management servers into cloud, or virtual, environments. Capacity planning for virtual environments typically involves the specification of sufficient physical resources to provide the illusion of unconstrained resources in an environment that may be characterized by highly variable demand In this document we will provide an IBM BigFix overview, including functionality, architecture, and performance. We will then offer the capacity planning recommendations, including considerations for hardware configuration, software configuration, and cloud best practices. A summary “cookbook” is provided to manage installation and configuration for specific instances of BigFix. Note: This document is considered a work in progress. Capacity planning recommendations will be refined and updated as new BigFix releases are available. While the paper in general is considered suitable for all BigFix Version 9.x releases, it is best oriented towards IBM BigFix Version 9.2.6 onwards, including IBM BigFix 9.5. In addition, a number of references are provided in the References section. These papers are highly recommended for readers who want detailed knowledge of BigFix server configuration, architecture, and capacity planning. Note: Some artifacts are distributed with this paper (see “View Navigation Panels Attachments” in the document viewer). The distributions are in zip format. However Adobe protects against files with a “zip” suffix. As a result, the file suffix is set to “zap” per distribution. To use these artifacts, simply rename the distribution to “zip” and process as usual.
11
2
IBM BigFix 9.x Overview An overview of IBM BigFix Version 9.x will be provided from the following perspectives: 1. Functional 2. Architectural 3. Return on Investment
2.1
Functional Overview The IBM BigFix portfolio provides a comprehensive security solution encompassing a number of operational areas. These areas include the following.
Lifecycle management (asset discovery and inventory, software distribution, patch management, operating system deployment, remote control).
Security and compliance (security configuration management, vulnerability management, patch management, anti-virus and anti-malware client management, network self-quarantine).
Patch management.
Power management.
Software use analysis.
Core protection.
Server automation.
Additional information on the functional management may be obtained from a variety of IBM resources (for example, this announcement letter for IBM BigFix 9.2: URL). Note that IBM BigFix was previously known as IBM Endpoint Manager, but was rebranded in 2015 with a very positive response from the field. In general, IBM BigFix spans the broadest OS and device set in the industry, including the management of physical and virtual servers, PCs, Macs, tablets, smartphones, embedded and hardened devices, and point of sale devices. This is managed via a scalable distributed infrastructure that includes a lightweight dedicated agent. We will describe this infrastructure in the architectural overview section.
12
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
2.2
Architectural Overview The following diagram provided a basic view of the BigFix architecture.
Figure 2: BigFix Architecture
The notable components of this diagram follow.
Root Server. The base BigFix Enterprise Server. It is comprised of a number of core services as identified.
Console. A management console (user interface) for BigFix operators. The console is a Windows only application. A console server is used to support one or more instances of the BigFix Console.
The WebUI. A Node.js instance with associated database intended to support the Web based user interface.
The Web Reports Server. The Web Reports Server can provide a variety of stock and custom reports for one or more BigFix server installations.
13
Relays. A distributed, hierarchical infrastructure to manage the deployment of BigFix agents across diverse network topologies.
Agents (as part of the client population). A lightweight, native agent that manages the endpoint.
Fixlet Servers (represented via the Internet content). These servers are used as the object repository for all client content (Fixlets, tasks, baselines, and analyses). In addition, dashboards, wizards, and WebUI applications are delivered via the Fixlet servers. Fixlets are utilized by the agent to determine relevance, compliance, and remediation activities.
Disaster Server Architecture (aka DSA, not shown). The DSA is a server replication architecture intended to provide fault tolerance.
The Database Management Server, or DBMS (either Microsoft SQL Server or IBM DB2 for Linux, UNIX, and Windows, also referred to as DB2 LUW).
The diagram below shows the anti-collocation options for these elements (meaning, the ability to deploy on nodes distinct from the BigFix root server). The pros and cons of anti-collocation are described later in this document.
Figure 3: BigFix Server Elements
Recommendations will be provided in the BigFix Capacity Planning section for optimal performance management of these components.
2.3
Return on Investment Return on Investment (ROI) is a key concern for any deployed solution. In the security space, the notion of “return” can involve many dimensions, given the potentially catastrophic impact security exploits may have on an enterprise. To facilitate the understanding of ROI for IBM BigFix, a business value analyst is available (URL).
14
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
Figure 4: Business Value Analyst for IBM BigFix and MobileFirst
The analyzer is based on the establishment of a comprehensive profile comprising the following elements.
The company profile, including legacy systems and current incident and problem resolution rates.
Hardware and software investment profiles, including endpoint audits and device decomposition.
Patch, application, and power management profiles.
The security management profile.
Based on these responses, the benefits, investments, and overall Return on Investment is provided through a number of multi-year views.
15
3
BigFix Performance Management We will provide an overview of BigFix performance management including reference benchmarks, key performance indicators, and monitoring tools.
3.1
Reference Benchmarks There are a number of reference benchmarks managed for the BigFix solution. These benchmarks ensure the offering is “field ready” and able to manage future scalability requirements. The set of benchmarks includes, but is not limited to, the following.
The concurrent user performance benchmark. A set of workloads to simulate user activity and responsiveness. The responsiveness is managed for both the console and Web User Interface offerings.
The client metering & evaluation loop benchmark. A set of workloads to manage client performance, including device impact and request latency.
FillDB benchmark. A set of workloads to manage and optimize the BigFix FillDB operation.
Mailbox benchmark. A set of workloads to manage and evaluate the BigFix endpoint mailbox functionality.
Relay plugins benchmark. A set of workloads to evaluate the replay plugin, including resource utilization and request latency.
REST API benchmark. A set of workloads to evaluate the RESTful interface of BigFix.
Web Reports benchmark. A set of workloads to evaluate the Web Reports interface of BigFix.
The Unified Benchmark. The comprehensive benchmark for BigFix. We will discuss the Unified Benchmark in more detail.
The most basic performance methodology for any benchmark is to establish a baseline, and then iterate on the baseline as you drive improvements through code, infrastructure, and tuning. Once the solution offering is delivered, the baseline is then moved to the new “improved” state.
3.1.1 The Unified Benchmark The Unified Benchmark combines the component level benchmarks, into a single, unified benchmark that provides comprehensive simulation and prediction of the behavior of BigFix in large customer environments. The benchmark is not only defined by the diversity
16
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
and scale of the workload, but also by time. Sample characteristics of the Unified Benchmark include the following.
Duration. The benchmark is persistent, meaning it is continuously running (as a customer workload would be continuously running) in order to manage the long term stability characteristics of BigFix (e.g. performance stability, system resource stability, etc.).
Data population. Database population is used to simulate large scale customer installations. Sample population parameters include, but are not limited to, the following. o
Number of managed devices = 250,000.
o
Number of managed Fixlets = 100,000 (20% custom).
o
Sites = 20 (25% custom sites).
o
Software packages = 750.
o
Patches = 28,000.
o
Users = 100 (10% master operators).
o
Roles = 50.
o
Computer groups= 50.
Client simulation. Real and simulated clients may be used to represent large installations comprising hundreds of thousands of endpoints.
Workload saturation. Workload levels should not be constant. Workload oscillation, meaning workload peaks and valleys are in evidence, are expected in customer environments. It can be useful to drive beyond solution saturation levels for brief periods to demonstrate product stability and preservation of service under high load.
User transaction rate control. The frequency that simulated users drive actions within the workload is managed via loop control functions. Closed loop simulation approaches are used where a new user will enter the system only when a previous user completes. Through the closed loop system, steady state operations under load may be driven. Sustained user transaction rates are characterized by a warm up phase, where users are introduced to the system in a controlled manner until the desired concurrency is reached. For example, in our tests, we expect to introduce a new user every five seconds as part of the warm up phase.
Think times. Think times are the “pause” between user operations, meant to simulate the behavior of a human user.
Bandwidth throttling. In order to simulate low speed or high latency lines, bandwidth throttling is employed for some customer workloads. A sample throttle for a moderate speed
17
ADSL connection (cable/DSL simulation) is a setting of 1.5 Mbps download, and 384 Kbps upload. The following diagram shows a sample unified benchmark environment.
Figure 5: BigFix Performance Benchmark Environment Sample
3.2
Key Performance Indicators Key Performance Indicators (KPI) are the quantitative values managed in the reference benchmarks that are used to determine solution success. Sample KPIs may include the following.
Request characteristics including response times and associated latency.
System utilization including the standard CPU, IO, network and memory views.
Request concurrency characteristics.
The final category, request concurrency, has many interesting dimensions. Two areas we will focus on are Little’s Law, and how to evaluate the number of concurrent users for a solution.
18
IBM BigFix Version 9.x: Capacity Planning, Performance, and Management Guide
3.2.1 Little’s Law The field of queuing theory is mathematically rich and often complex. However, Little’s Law offers a simple and intuitive view of queuing theory. Little’s Law may be summarized by the following figure.
L = λW Where: L = the number of concurrent requests in the system. λ = the request arrival rate. W = the average time a request spends in the system. Figure 6: Little's Law
This elegant equation makes it clear that if you want to improve concurrency you may:
Increase the request arrival rate.
Reduce the request service time.
Increasing the arrival rate will eventually hit a solution limit (whether software or infrastructure). At that point, the focus is typically on optimizing the software and/or infrastructure to reduce the average request time.
3.2.2 Evaluating the Number of Concurrent Users The number of concurrent users supported by a solution is typically a function of request response times under load. For example, if a solution can manage 50 concurrent users with an average response time of “X” seconds, it comes down to whether “X” is acceptable for the user (where the user may be a person, a program, etc.). For the concurrent users of a user interface (e.g. the BigFix console or WebUI), it is important to understand what is meant by a concurrent user. Consider:
P = total population for an instance of BigFix (including administrators, end users, etc.).
C = the concurrent user population for an instance of BigFix. Concurrent users are considered to be the set of users within the overall population P that are actively managing the environment at a point in time (e.g. administrator operations in the user interface, endpoint operations, etc.).
In general, P is a much larger value than C (i.e. P >> C). For example, it is not unrealistic that a total population of 200 users may have a concurrent user population of 20 users (i.e. 10%).
19
3.3
Monitoring Tools Monitoring tools may include system monitoring approaches as well as associated infrastructure benchmarks. The follow table describes a number of approaches that are used for BigFix. Tool
Description
BigFixPerf
A BigFix custom data collection tool that wraps the nmon and perfmon utilities. It is a Perl based utility and requires an instance of Perl. Documentation: See detail section below. Recommended invocation: BigFixPerf –monitor
–interval –iterations
BigFixDisk
A BigFix custom data collection tool that logs disk utilization. It is a Perl based utility and requires an instance of Perl. Documentation: See detail section below. Recommended invocation: BigFixDisk –monitor –interval –iterations
nmon
nmon is a comprehensive system monitoring tool for the UNIX platform. It is highly useful for understanding system behavior. Documentation: URL Sample invocation: nmon -T -s -c -F 
