Transcript
Lots of Stuff Gordon D. Brown Science & Technology Facilities Council WLCG Service S i Reliability R li bili W Workshop k h CERN 26-30 November 2007
Overview • • • • • •
Database Security Passwords Database Security Checklist Host Housekeeping Grid Control Backups Overview
• Interactive • Any questions
WLCG Service Reliability Workshop
2
Database Security
WLCG Service Reliability Workshop
3
Database Security • Authenticating DBA logins on host – Operating system (o/s) authentication – A password file
• For nonsecure remote connections – Must be authenticated by a password file
• Connecting to database as a privileged user over a local connection or a secure remote connection in two ways – If have password file and have been granted the SYSDBA or SYSOPER system privilege – If no password file, then operating system authentication, username off the th database d t b administrator d i i t t iin a special i l group • OSDBA:. Users in that group are granted SYSDBA privileges group p are granted g SYSOPER privileges p g • OSOPER: Users in that g WLCG Service Reliability Workshop
4
Database Security • Operating System Authentication – OSDBA and OSOPER – The groups are created and assigned specific names as part of the database installation process. – If you are not a member of either of these operating system groups and you attempt to connect as SYSDBA or SYSOPER, the CONNECT command fails
WLCG Service Reliability Workshop
5
Database Security • Using Password File Authentication – To enable authentication of an administrative user using password file authentication you must do the following: – If not already created, create the password file using the ORAPWD utility: • ORAPWD FILE=filename PASSWORD=password p ENTRIES=max_users
• For RAC the environment variable for each instance should point to the same password file • Protect p password file!
WLCG Service Reliability Workshop
6
Database Security • Operating p g system y authentication takes p precedence over password file authentication. • S Specifically, ifi ll if you are a member b off the h OSDBA or OSOPER group for the operating system, and you connect as SYSDBA or SYSOPER, you will be connected with associated administrative privileges regardless of the username/password that you specify. i.e. sqlplus / as sysdba • If you are not in the OSDBA or OSOPER groups, and you are not in the password file, then attempting to connect as SYSDBA or as SYSOPER fails.
WLCG Service Reliability Workshop
7
Database Security • Setting REMOTE_LOGIN_PASSWORDFILE – With password file, you must set the initialization parameter REMOTE_LOGIN_PASSWORDFILE. Values: • NONE: Database behaves as if password file does not exist. – No privileged connections are allowed over nonsecure connections.
• EXCLUSIVE: (default) Used with only one instance of one database. – Only an EXCLUSIVE file can be modified – Enables you to add, modify, and delete users. It also enables you to change the SYS password with the ALTER USER command.
• SHARED: Used by multiple databases running on the same server, or multiple instances of a (RAC) database. – Password file cannot be modified. – This option is useful if you are administering multiple databases or a RAC database. WLCG Service Reliability Workshop
8
Database Security • Login overview
WLCG Service Reliability Workshop
9
Database Security • Viewing Password d File l Members b – V$PWFILE_USERS view to see the users who have been granted SYSDBA or SYSOPER • USERNAME: User that is recognized by the password file • SYSDBA: If TRUE, then the user can log on with SYSDBA • SYSOPER: if TRUE, then the user can log on with SYSOPER
WLCG Service Reliability Workshop
10
Database Security • Ensure that internal networks are inside a firewall • Oracle Advanced Security – Add on – Enables data encryption and integrity checking, enhanced authentication, single sign-on, and support for DCE – Combines network encryption, database encryption and strong t authentication th ti ti together t th to t help h l customers t address dd privacy and compliance requirements:
WLCG Service Reliability Workshop
11
Database Security • Transparent Data Encryption – Protection of stored data by transparently encrypting data (using 3DES or AES with up to 256 bits) at the column or tablespace level.
• Network encryption and data integrity – Supports: • • • •
RC4 (40 (40, 56 56, 128 128, and 256 bits) DES (40 and 56 bits) 3DES (2 and 3 keys) AES (128, 192, and 256 bits)
WLCG Service Reliability Workshop
12
Database Security • Strong authentication – Two-factor (or "strong") authentication is based on something the user has (a smart card, token, etc.) and a PIN or passcode. Supports: • • • •
Kerberos RADIUS (Remote Authentication Dial Dial-In In User Service) Secure Sockets Layer (with digital certificates) PKI
WLCG Service Reliability Workshop
13
Database Passwords • • • •
Use strong g passwords p Try against a password checker Passwords stored as hashes in database Cleartext passwords can be typically but not necessarily found at the following places – – – – – – –
Server Shell History files Unix Scripts Log Files Dump Files Trace Files Application Server
– – – – – – –
JDBC-Config-Files Trace Files DBA Client PC Desktop Shortcut Desktop-Shortcut Batch-Files Tool Configuration files (connections.ini) Trace Files
WLCG Service Reliability Workshop
14
Database Passwords • Do not email passwords around • Where are yours written down? • Do your machine room staff have them? In emergency? • Who actually knows the passwords? A list of people? • Who has access to the host (as root? oracle?) • Credentials C d i l often f iin G Grid id C Controll
WLCG Service Reliability Workshop
15
Database Passwords • We use “pwsafe” – – – –
Use SSH key to access Password on pwsafe to use File l replicated l d across two sites Usernames and passwords are stored using a unique name to identify them, normally of the form group.name. However the group name is optional. • o/s
. • database d b users: . • application express: .
WLCG Service Reliability Workshop
16
Database Passwords
• pwsafe – Showing A Password: • # pwsafe up
– Adding A Password: • # pwsafe a
– Deleting A Password: • # pwsafe delete
• Can add notes too WLCG Service Reliability Workshop
17
Machine Ports • Check the Oracle default port list • Changing the default ports can help to stop simple attacks but not real portscans. • In Oracle it is very often not possible to change the default port because the port is hardcoded. • At least for the Oracle database (except iasdb) it's is recommended to change the TNS listener port from 1521/1526 to something else. • Options include running a local firewall then using something called SQLNet Proxy to manage the port. Otherwise we could use Connection Manager.
WLCG Service Reliability Workshop
18
Database Security Checklist
WLCG Service Reliability Workshop
19
Database Security Checklist – Step 1 • Install only onl what hat is required req ired • Oracle Database Installation contains a host of options and products in addition to the database server. necessary. • Install additional products and options only as necessary • Use the custom installation option to avoid installing unnecessary products d t or, perform f a ttypical i l iinstallation, t ll ti and d then deinstall unrequired options and products. • It is best practise to know what the database will be used for so that you can apply only what Oracle products are needed.
WLCG Service Reliability Workshop
20
Database Security Checklist – Step 1 • Common components that you can install with each database are: – – – – – – – – –
Oracle Data Mining Oracle Text Oracle Olap O l Spatial Oracle S i l Oracle Ultra Search Oracle Label Security Sample Schemas Enterprise Manager Repository Oracle database extensions for .NET
WLCG Service Reliability Workshop
21
Database Security Checklist – Step 1 • N None off these h are needed d d to get a database d b up and d running. i They are "extras" to be added for more functionality. • Check which of these have been installed by running DBCA • ACTION: Use the instructions below to deinstall any options on PRODUCTION databases that are not actually used – Dropping Sample Schemas – Sample Schemas script directories are located in $ORACLE_HOME/demo/schema. – The script p xx_drop.sql, p q , where xx is the schema abbreviation,, removes all objects from a particular schema. Run this for HR/OE/PM/IX/SH/BI schemas.
WLCG Service Reliability Workshop
22
Database Security Checklist – Step 1 • Dropping schemas (cont) # cd $ORACLE_HOME/demo/schemas # sqlplus hr/pass SQL> @human_resources/hr_drop SQL> exit # sqlplus oe/pass SQL> @order_entry/oe_drop SQL exit SQL> it # sqlplus pm/pass SQL> @product_media/pm_drop SQL> exit
WLCG Service Reliability Workshop
23
Database Security Checklist – Step 1 • Dropping schemas (cont) # sqlplus ix/pass SQL> @info_exchange/dix_v3.sql -- ignore any errors SQL> exit # sqlplus sh/pass SQL> @sales_history/sh_drop.sql -- ignore i any errors SQL> exit
WLCG Service Reliability Workshop
24
Database Security Checklist – Step 1 • Dropping schemas (cont) • no script to run for BI • O Once ce tthis s has as been bee do done e for o a all tthe e sc schemas, e as, d drop op tthe e users HR/OE/PM/IX/SH/BI. SQL> drop user cascade;
• Manual de-install of Spatial • Spatial is installed by default when using DBCA. To de deinstall manually follow Metalink Note:179472.1
• Al Also drop d th the MDDATA user (th (the notes t miss i thi this one) WLCG Service Reliability Workshop
25
Database Security Checklist – Step 2 • Lock and Expire Default User Accounts • Oracle installs a number of default database server user accounts. • On DBCA installation – most default database user accounts automtically locked and expireed p .
• On MANUAL installation – no accounts are locked
• The h following f ll two slides l d show h a list l off Oracle l account and d their status that is given AFTER the database is created using DBCA • ACTION: Check these accounts are still expired and locked if not being used. WLCG Service Reliability Workshop
26
Database Security Checklist – Step 2
WLCG Service Reliability Workshop
27
Database Security Checklist – Step 2
WLCG Service Reliability Workshop
28
Database Security Checklist – Step 3 • Review all database users every y 6 months (or ( more often)) • Contact users and check schemas are still required – If not export and drop them
• Iff any further f options or Oracle products e.g. Apex, Oracle Warehouse Builder, HTTP server have been installed, check they are still required – If not deinstall them
• Run the Oracle Installer in $ORACLE HOME/oui/bin/runInstaller to check if there are any $ORACLE_HOME/oui/bin/runInstaller other Oracle Homes with products in them and deinstall from there. • Other products like Oracle Warehouse Builder may have an alternative de-install process. Check the documentation for your p y product. WLCG Service Reliability Workshop
29
Database Security Checklist – Step 4 • Change default User Passwords • Unlock ONLY those accounts that need to be accessed on a regular basis and assign a strong password to each of these accounts. t – Minimum password of 8 alphanumeric characters. passwords are different between the users – Make sure the p – Make sure they are also different for any counterpart account on a production/development/test account.
• This is mandatory for all DBA accounts on the database – sys, system, sysman, dbsnmp
WLCG Service Reliability Workshop
30
Database Security Checklist – Step 4 • Change passwords with $ sqlplus / as sysdba SQL> alter user system identified by ********; User altered. SQL>
• B Basic i password d management rules l ((such h as llength, h hi history, complexity etc...) as provided by the database be applied to all user passwords and that these passwords are changed periodically. • ACTION: ensure all passwords are at least 8 random alphanumeric characters. If change of password is required to satisfy this, make sure you advise users/arrange appropriate downtime as an application pp may y need to change g it's settings. g WLCG Service Reliability Workshop
31
Database Security Checklist – Step 4 • ACTION: Change SYS SYS, SYSTEM SYSTEM, SYSMAN (on Grid Control DB), DB) DBSNMP passwords every 6 months. Remember to change preferred credentials in Grid Control, agent config and pwsafe (or your wherever you store passwords) passwords). • Changing DBSNMP password – Change DBSNMP password as above. To tell the agent that the password has changed, edit the file $AGENT_HOME//sysman/emd/targets.xml and edit the database target g section as follows: Property NAME="UserName" VALUE="dbsnmp"; Property NAME="password" VALUE="newpass" ENCRYPTED="FALSE";
• Then stop and start the agent. The password will be encrypted again when the agent restarts. WLCG Service Reliability Workshop
32
Database Security Checklist – Step 5 • Enable Data Dictionary Protection • To prevent regular users that have ANY system privileges e.g. DROP ANY TABLE being able to use these on the data di i dictionary make k sure the h parameter iis O7_DICTIONARY_ACCESSIBILITY is set to FALSE. • Action FALSE is the default in 10g but check it anyway $ sqlplus / as sysdba SQL> show parameter O7_DICTIONARY_ACCESSIBILITY NAME TYPE VALUE ------------------------------- ------------ ----07_DICTIONAIRY_ACCESSIBILITY boolean FALSE
WLCG Service Reliability Workshop
33
Database Security Checklist – Step 5 • If a user does d need d to access a di dictionary i view i ffor non malicious information, they can be granted the "SELECT ANY DICTIONARY" privilege.
WLCG Service Reliability Workshop
34
Database Security Checklist – Step 6 • Practise the principle of least privilege • Do not provide users with any more priviliges than are strictly necessary • Restrict the following as much as possible: – The number of SYSTEM and OBJECT privileges granted to database users – 2) The number of people who are allowed to make SYS-privileged connections to the database.
• ACTION: Review the SYSTEM privs (Note most regular users should only have CREATE SESSION privilege or CONNECT role assigned d ((although lh h check h k CONNECT CO C role l only l contains "CREATE SESSION" privilege in DBA_SYS_PRIVS).
WLCG Service Reliability Workshop
35
Database Security Checklist – Step 6 • ACTION: The RESOURCE role (allows creation of tables etc) should only be assigned on creation of the schema and for schema changes and then dropped again on production databases) Drop RESOURCE role from regular users (see databases). DBA_SYS_PRIVS). Review the contents of the RESOURCE role in DBA_ROLE_PRIVS. • ACTION: Also "ANY" privileges should not be assigned as they allow this user access to objects in all other schemas. Change ANY system privileges to object privileges e.g. drop SELECT ANY TABLE and assign ....
WLCG Service Reliability Workshop
36
Database Security Checklist – Step 6 • To T check h k currently l assigned i d system privs i SELECT * from DBA_SYS_PRIVS WHERE GRANTEE NOT IN ('DBA', 'SYS', 'SYSTEM', 'OEM_MONITOR', 'OEM_ADVISOR', 'IMP_FULL_DATABASE', 'EXP_FULL_DATABASE', 'ORDSYS', 'ORDPLUGINS', 'JAVADEBUGPRIV', 'OUTLN', 'FLOWS_020200', 'FLOWS_030000', 'AQ_ADMINISTRATOR_ROLE', 'CTXSYS', 'EXFSYS', 'XDB', 'APEX_PUBLIC_USER', 'CONNECT', 'RESOURCE', 'DIP', 'SCHEDULER_ADMIN', _ 'DBSNMP', 'ANONYMOUS', 'WMSYS', 'RECOVERY_CATALOG_OWNER') ORDER BY GRANTEE, PRIVILEGE;
WLCG Service Reliability Workshop
37
Database Security Checklist – Step 6 • Review R i and d revoke k unneccessary privs i – GRANT CONNECT TO ; – REVOKE CREATE PROCEDURE FROM ; – REVOKE RESOURCE FROM ;
• To see roles assigned to users – SELECT * FROM dba_role_privs WHERE USER = 'ADAM';
• ACTION ACTION: Review R i object bj privs i ffor each h regular l user and d revoke k as necessary – select * from dba_tab_privs where grantee ='SSO';
• ACTION: Check that only SYS has DBA role
WLCG Service Reliability Workshop
38
Database Security Checklist – Step 6 • Addi Additionally, i ll review i the h privileges i il grants to the h PUBLIC role l as you may wish to restrict what access this will give to all users. • Oracle provides execution rights to four packages on the PUBLIC role that should be removed after installation: • ACTION: Revoke the execution privilege on PUBLIC to the p packages: g – – – –
UTL_SMTP [EMAIL] UTL_TCP [Network Connections] UTL HTTP [request/retrieve information from HTTP server] UTL_HTTP UTL_FILE [access to files outside the DB]
WLCG Service Reliability Workshop
39
Database Security Checklist – Step 6 • Remove R with ih # sqlplus / as sysdba SQL> REVOKE EXECUTE ON sys.utl y _smtp p FROM "PUBLIC"; Revoke succeeded. SQL> REVOKE EXECUTE ON sys sys.utl_tcp utl tcp FROM "PUBLIC" "PUBLIC"; Revoke succeeded. SQL> REVOKE EXECUTE ON sys.utl_http FROM "PUBLIC"; Revoke succeeded. SQL> REVOKE EXECUTE ON sys.utl_file FROM "PUBLIC"; Revoke succeeded. WLCG Service Reliability Workshop
40
Database Security Checklist – Step 7 • Enforce access controls effectively and authenticate clients stringently • S Setting tti th the iinitialization iti li ti parameter t REMOTE_OS_AUTHENT = FALSE, creates a more secure configuration that enforces proper, server-based authentication of clients connecting to an Oracle database. • You should not alter the default setting of the REMOTE_OS_AUTHENT initialization parameter, which is FALSE. Setting this parameter to FALSE does not mean that users cannot connect remotely remotely. It simply means that the database will not trust that the client has been already authenticated, and will therefore apply its standard authentication processes. WLCG Service Reliability Workshop
41
Database Security Checklist – Step 7 • ACTION ACTION: Check Ch k parameter t REMOTE_OS_AUTHENT is i FALSE on each database instance (all on RAC): $ sqlplus / as sysdba SQL> show parameter remote_os_authent NAME ---------------------remote_os_authent
TYPE VALUE --------------------- ----boolean FALSE
WLCG Service Reliability Workshop
42
Database Security Checklist – Step 8 • R Restrict i O Operating i S System A Access • Limit the number of operating system users. • Limit the privileges of the operating system accounts (administrative, root-privileged or DBA) on the host (physical machine) to the least privileges needed for the user's tasks. • ACTION: Check with System Administrator that only default users have been created and any y unused default ones are locked. Ask them to remove any old users. • ACTION ACTION: Check Ch k with i h System S Administrator Ad i i that h the h privileges i il of OS accounts have the least privileges necessary.
WLCG Service Reliability Workshop
43
Database Security Checklist – Step 8 • ACTION ACTION: Check Ch k that h any database d b d developers l h have their h i own O/S id and are in the osinstall group but not the dba group. This allows them to access sqlplus but not to log on as SYSDBA. • Restricting the ability to modify the default file and directory permissions for the Oracle Database home (installation) directory or its contents. Even privileged operating system users and d the h O Oracle l owner should h ld not modify dif these h permissions, unless instructed otherwise by Oracle.
WLCG Service Reliability Workshop
44
Database Security Checklist – Step 8 • ACTION ACTION: Check Ch k with i h System S Administrator Ad i i that h no O/S users can change file/directory permissions of the Oracle Home • Restricting symbolic links. Ensure that when providing a path or file to the database, neither the file nor any part of the path is modifiable by an untrusted user. user • The file and all components of the path should be owned by the DBA or some trusted account, such as root. This recommendation d ti applies li tto all ll ttypes off fil files: d data t fil files, llog files, trace files, external tables, bfiles, and so on. • ACTION: Check no symbolic links against data files, log files, trace files, external tables, bfiles etc
WLCG Service Reliability Workshop
45
Database Security Checklist – Step 9 • Restrict Network Access • Use a firewall • Oracle Listener (9i) – An Oracle Listener running without an established password may be probed for critical details about the databases on which it is listening such as trace and logging information, banner information and database descriptors and service names. – Restrict the privileges of the listener listener, so that it cannot read or write files in the database or the Oracle server address space. This restriction prevents external procedure agents spawned by the listener (or procedures executed by such an agent) from inheriting the ability to do such reads or writes.
WLCG Service Reliability Workshop
46
Database Security Checklist – Step 9 • Oracle Listener – Prevent online administration by requiring the administrator to have write privileges on the LISTENER.ORA file – Use the parameter ADMIN_RESTRICTIONS_listener_name to restrict runtime administration of the listener. The parameter is useful if the listener is not password-protected. – Setting ADMIN_RESTRICTIONS_listener_name=on disables the runtime modification of parameters in listener.ora. That is, the listener will refuse to accept SET commands that alter its parameters. – To change any of the parameters in listener.ora, including ADMIN_RESTRICTIONS_listener_name itself, modify the listener.ora file manually and reload its parameters (with the RELOAD command) for the new changes to take effect without explicitly stopping and restarting the listener. WLCG Service Reliability Workshop
47
Database Security Checklist – Step 9 • ACTION: Add or alter this line in the LISTENER LISTENER.ORA ORA file – ADMIN_RESTRICTIONS_LISTENER=ON
Then RELOAD the configuration. • If you are administering the listener remotely over an insecure network and require maximum security, security configure the listener with a secure protocol address that uses the TCP/IP with SSL protocol. If the listener has multiple protocol addresses, ensure that h the h TCP/IP with i h SSL protocoll address dd iis li listed d fi first iin the h listener.ora file.
WLCG Service Reliability Workshop
48
Database Security Checklist – Step 9 • ACTION: Use SSL when administering the listener, listener by making the TCPS protocol the first entry in the address list as follows: LISTENER= (DESCRIPTION= (ADDRESS_LIST= ( (ADDRESS= (PROTOCOL=tcps) (HOST = ed-pdsun1.us.oracle.com) (PORT = 8281)))
WLCG Service Reliability Workshop
49
Database Security Checklist – Step 9 • For Oracle Database 10g Release 1 and higher the default authentication mode is local O/S authentication which requires administrator to be a member of the local dba group. • Setting a password for the TNS listener in Oracle Database 10g Release 1 and higher simplifies administration. • However, However setting a password requires good password management to prevent unauthorized users from guessing the password and potentially gaining access to privileged listener operations. • Customers C t may wish i h to t consider id nott setting tti a password d ffor th the TNS listener starting with Oracle Database 10g Release 1. • ACTION ACTION: No N need d tto sett a li listener t password d ffor release l 10 10.1 1 or higher but ensure that only the oracle O/S user (or equivalent) is a member of the dba O/S group (or equivalent) WLCG Service Reliability Workshop
50
Database Security Checklist – Step 9 • ACTION: Remove the external procedure configuration from the listener.ora file if you do not intend to use such procedures. Remove the following EXTPROC_LISTENER= (DESCRIPTION (DESCRIPTION= (ADDRESS= (PROTOCOL=ipc)(KEY=extproc)))
WLCG Service Reliability Workshop
51
Database Security Checklist – Step 9 • Monitor listener activity • Authenticating client computers over the Internet is problematic problematic. Do user authentication instead, which avoids client system issues that include falsified IP addresses, hacked operating systems or applications, li i and d ffalsified l ifi d or stolen l client li system id identities. ii SSL. Using SSL (Secure Sockets • Configure the connection to use SSL Layer) communication makes eavesdropping unfruitful and enables the use of certificates for user and server authentication. (requires OAS??)
WLCG Service Reliability Workshop
52
Database Security Checklist – Step 9 • Consider setting up certificate authentication for clients and servers such that: – The organization is identified by unit and certificate issuer and the user is identified by distinguished name and certificate issuer. – Applications test for expired certificates. – Certificate revocation lists are audited
WLCG Service Reliability Workshop
53
Database Security Checklist – Step 9 • ACTION: Check network IP addresses – Use the Oracle Net valid node checking security feature to allow or deny access to Oracle server processes from network clients with specified ifi d IP addresses. dd T To use thi this feature, f t sett the th following f ll i protocol.ora (Oracle Net configuration file) parameters: tcp.validnode_checking = YES tcp.excluded_nodes = {list of IP addresses} tcp.invited nodes = {list of IP addresses} tcp.invited_nodes – The first parameter turns on the feature whereas the latter parameters respectively i l d deny and d allow ll specific ifi client li IP addresses dd ffrom making ki connections to the Oracle listener (This helps in preventing potential Denial of Service attacks). WLCG Service Reliability Workshop
54
Database Security Checklist – Step 9 • Encrypt network traffic – If possible, use Oracle Advanced Security to encrypt network traffic between clients, databases, and application servers.
• Harden the operating system – disabling all unnecessary operating system services services. – Both UNIX and Windows platforms provide a variety of operating system services, most of which are not necessary for most deployments Such services include FTP, deployments. FTP TFTP, TFTP TELNET TELNET, and so forth forth. – Be sure to close both the UDP and TCP ports for each service that is being disabled. Disabling one type of port and not the other does not make the operating system more secure secure. – See your friendly local system administrator
WLCG Service Reliability Workshop
55
Database Security Checklist – Step 10 • Apply all security patches • Always l apply l all ll relevant l and d current security patches h ffor b both h the operating system on which Oracle Database resides and Oracle Database itself, and for all installed Oracle Database options and components. • Periodically check the security site on Oracle Technology Network for details on security alerts released by Oracle Corporation at h http://www.oracle.com/technology/deploy/security/alerts.htm // l / h l /d l / i / l h
WLCG Service Reliability Workshop
56
Database Security Checklist – Step 10 • Also check Oracle Worldwide Support Service site, Metalink, for details on available and upcoming security-related patches att http://metalink.oracle.com htt // t li k l • ACTION: Check with System Administrator re outstanding OS security patches and apply • ACTION: Install Oracle Critical Patch updates as soon as possible after release
WLCG Service Reliability Workshop
57
Database Security Checklist – Step 11 • Contact Oracle Security products if you come across a vulnerability in Oracle Database • If you believe that you have found a security vulnerability in Oracle Database, then submit a Service Request to Oracle Worldwide Support Services using Metalink, or e-mail a complete description of the problem, including product version and platform, platform together with any exploit scripts and examples to the following address: [email protected]
WLCG Service Reliability Workshop
58
Database Security Checklist – Other Steps • Policies in Enterprise Manager – Each target should be checked to make sure it complies with all security polices for host, database and cluster instances, listener and other targets where appropriate.
• ACTION: Go to Grid Control, Control click on Compliance | Policy Violations. Correct any security violations for your system including Host, Cluster Database, Database Instance and Listener targets. Li
WLCG Service Reliability Workshop
59
Database Security Checklist – Other Steps • Personal Data (from HBI audit) – If the database schema contains Personal data it must comply to th Data the D t Protection P t ti Act. A t – Define Personal Data - what are the rules of compliance?
• Sensitive Data (from HBI audit) – If the database schema contains Sensitive data it must comply to the Data Protection Act. Act – Define Sensitive Data - what are the rules of compliance?
WLCG Service Reliability Workshop
60
Database Auditing • Audit any object NAME ---------------audit_file_dest audit_sys_operations audit_trail
VALUE -----/opt/oracle/app/oracle/admin/ngsdb/adump FALSE DB
• See dba_audit_* tables • There is an AUDIT privilege • Logins/logouts stored in dba_audit_session
WLCG Service Reliability Workshop
61
Database Auditing • Check logins SET LINESIZE 120; COL username FORMAT a15 COL terminal t i l FORMAT a15 15 COL logon_time FORMAT a25 COL logoff_time FORMAT a25 COL action_name FORMAT a20 COL returncode FORMAT 9999 SELECT username, terminal, action name action_name, TO_CHAR(timestamp,'DD-MON-YYYY HH24:MI:SS') logon_time, TO_CHAR(logoff_time,'DD-MON-YYYY HH24:MI:SS') logoff_time, returncode FROM dba_audit_session WHERE ROWNUM < 16 ORDER BY logoff_time DESC;
WLCG Service Reliability Workshop
62
Database Auditing • Check logins
WLCG Service Reliability Workshop
63
Database Auditing • O/S logins monitored in ora ora_.aud aud files Audit file /opt/oracle/app/oracle/admin//adump/ora_22155.aud Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options ORACLE_HOME = /opt/oracle/app/oracle/product/10.2.0 System name: Linux Node name: Release: 2.6.9-55.0.2.ELsmp Version: #1 SMP Tue Jun 12 17:59:08 EDT 2007 Machine: i686 Instance name: Redo thread mounted by this instance: 1 Oracle process number: 76 Unix process pid: 22155, image: oracle@ (TNS V1-V3) Wed Nov 28 20:54:00 2007 ACTION : 'CONNECT' DATABASE USER: '/' PRIVILEGE : SYSDBA CLIENT USER: oracle CLIENT TERMINAL: pts/0 STATUS: 0
WLCG Service Reliability Workshop
64
Host Security • Use ssh key pairs – Update /etc/ssh/sshd_config where you can say that you do not want to allow password authentication PasswordAuthentication yes to
PasswordAuthentication no then restart the ssh daemon /etc/init.d/sshd restart
– After that change you need to have your public ssh key in the ~/.ssh/authorized_keys file for any user that you would like to log g in as.
WLCG Service Reliability Workshop
65
Other Security • Do not leave host names or database names in presentations etc • Google your hosts (or will that then go in their cache?) • Lock your computer when you’re away • Don’t leave p putty y or “screen” sessions open p • Speak to your security officer if in any doubt • Arrange a security audit
WLCG Service Reliability Workshop
66
Host Housekeeping M Managing i Y Your E Environment i t
WLCG Service Reliability Workshop
67
Host Housekeeping • Log files can grow to large sizes and be difficult to view or take up too much space • We have developed script to purge them – Runs R through h h Grid G id Control C l
• Make sure files are backed up p first if y you need them • Balance between history and space
WLCG Service Reliability Workshop
68
Host Housekeeping function rmfiles { DIR $1 DIR=$1 DESC=$2 DAYS=$3 if [ -d $DIR ] then echo ` ` echo "Checking ${DESC}: ${DIR}" echo "Removing files/directories older than than: $DAYS days" # The * stops find deleting the directory itself echo `find ${DIR}/* -mtime +$DAYS | wc -l` files or directories # This lists the files/dirs to delete find ${DIR}/* -mtime +$DAYS # then deletes them find ${DIR}/* -mtime +$DAYS -exec rm -rf {} \; else echo "Directory $DIR does not exist on this server" fi } WLCG Service Reliability Workshop
69
Host Housekeeping echo "Housekeeping files" date +\%d\%m\%y_\%H\%M\%S echo `date` date OBASE=/opt/oracle/ora01/app/oracle OHOME=$OBASE/OracleHomes/10.2.0/db $ / / / _1 AGHOME=/opt/oracle/ora01/app/oracle/OracleHomes/oms10g/agent10g # For RAC CRSHOME=/opt/oracle/crs/oracle/product/10/app # For HTTP server installations HTTPHOME=/opt/oracle/ora01/app/oracle/OracleHomes/oms10g/oms10g # For Grid Control installations OMSHOME=/opt/oracle/ora01/app/oracle/OracleHomes/oms10g/oms10g
WLCG Service Reliability Workshop
70
Host Housekeeping rmfiles "$OBASE/admin/$ORACLE_SID/bdump" "BACKGROUND dump destination" 120 rmfiles "$OBASE/admin/$ORACLE_SID/cdump" "CORE dump destination" 31 rmfiles "$OBASE/admin/$ORACLE_SID/dpdump" "DATA PUMP destination" 31 rmfiles "$OBASE/admin/$ORACLE_SID/udump" "USER dump destination" 31 rmfiles "$OBASE/admin/$ORACLE_SID/adump" "AUDIT destination" 31 rmfiles "$OBASE/admin/$ORACLE_SID/hdump" "HIGH AVAILABILITY tracefile destination" 31 rmfiles "$OHOME/admin/+ASM/bdump" "ASM BACKGROUND dump destination" 31 rmfiles "$CRSHOME/log/`hostname -s`/client" "CRS client logs" 31 rmfiles "$AGHOME/`hostname`/sysman/log" $AGHOME/ hostname /sysman/log "AGENT AGENT logs logs" 31 rmfiles "$AGHOME/sysman/log" "AGENT logs" 31 rmfiles "$HTTPHOME/Apache/Apache/logs" "HTTP SERVER logs" 31 rmfiles "$OMSHOME/opmn/logs" $ / p / g "OMS opmn p logs" g 31 rmfiles "$OMSHOME/sysman/log" "OMS sysman logs" 31
WLCG Service Reliability Workshop
71
Host Housekeeping Housekeeping files 291107_074653 Thu Nov 29 07:46:53 GMT 2007 Checking BACKGROUND dump destination: /opt/oracle/ora01/app/oracle/admin/mercury/bdump Removing files/directories older than: 120 days 1 files or directories /opt/oracle/ora01/app/oracle/admin/mercury/bdump/alert_mercury.log.200 70731 Checking CORE dump destination: /opt/oracle/ora01/app/oracle/admin/mercury/cdump Removing files/directories older than: 31 days find: i / /opt/oracle/ora01/app/oracle/admin/mercury/cdump/*: / / / / / i / / / No such file i or directory 0 files or directories . . Completed Thu Nov 29 07:46:55 GMT 2007
WLCG Service Reliability Workshop
72
WLCG Service Reliability Workshop
73
WLCG Service Reliability Workshop
74
WLCG Service Reliability Workshop
75
WLCG Service Reliability Workshop
76
WLCG Service Reliability Workshop
77
WLCG Service Reliability Workshop
78
Alert Log Maintenance • Script keeps alert log to a managable size • Allows file to be emailed daily #!/bin/bash BDUMP=/opt/oracle/app/oracle/admin/ogma/bdump # This script will move and rename the current alert.log by adding the date as the extension # GDB 08-OCT-2004 # Email the log cat $BDUMP/alert_ogma1.log | mail -s "Oracle: ogma1 alert log for `date`" db_admins # Rename the log CURRDATE=`date '+%Y%m%d'` cp $BDUMP/alert_ogma1.log $BDUMP/alert_ogma1.log.$CURRDATE cat /dev/null > $BDUMP/alert_ogma1.log WLCG Service Reliability Workshop
79
Alert Log Maintenance • Run daily from cron 0 8 * * * /home/oracle/mercury/alert.sh >/dev/null 2>&1
• Emailed to group – Details D t il iin /home/oracle/.mailrc /h / l / il fil file alias db_admins [email protected]
WLCG Service Reliability Workshop
80
Alert Log Maintenance • Keeps alert logs small and easy to find -rw-r-----rw-r-----rw-r-----rw-r-----rw-r-----rw-r-----rw-r-----rw-r-----rw-r-----rw-r-----
1 1 1 1 1 1 1 1 1 1
oinstall oinstall i ll oinstall oinstall oinstall oinstall oinstall oinstall oinstall oinstall
35347 35449 35654 35974 36269 36539 34669 35014 35465 0
Nov Nov N Nov Nov Nov Nov Nov Nov Nov Nov
20 21 22 23 24 25 26 27 28 28
08:00 08:00 08 00 08:00 08:00 08:00 08:00 08:00 08:00 08:00 08:00
alert_ogma1.log.20071120 alert_ogma1.log.20071121 l 1 l 20071121 alert_ogma1.log.20071122 alert_ogma1.log.20071123 alert_ogma1.log.20071124 alert_ogma1.log.20071125 alert_ogma1.log.20071126 alert_ogma1.log.20071127 alert_ogma1.log.20071128 alert_ogma1.log
WLCG Service Reliability Workshop
81
Host Housekeeping • Directory Structure – – – – – – – – – –
/opt/oracle/product/10.2.0/db_1 # oracle database home /opt/oracle/product/10.2.0/client_1 # oracle client home /opt/oracle/product/10.2.0/agent_1 # oracle agent home /opt/oracle/product/10.2.0/crs # oracle clusterware home /opt/oracle/oraInventory # oracle inventory /opt/oracle/archive/ # archives if not using ASM /opt/oracle/backup/ # backups / /opt/oracle/oradata/ / l / d / d #d data iff not using ASM /opt/oracle/admin/ # logs and alerts //opt/oracle/flash_recovery_area/ p/ / y / # flash recovery y area if used
WLCG Service Reliability Workshop
82
Host Housekeeping • All database, database environment and schema scripts should be kept under /home/oracle • The /home/oracle directory should be the home directory for the unix user oracle • In /home/oracle the following scripts should be available: – ora10 name>.sh sh # Script to set database SID – ora10asm.sh # Script to set ASM SID – ora10common.sh # Script to set common environment variables
• Scripts should use LOGIN_PATH and LOGIN_LD_LIBRARY_PATH environment variables so that ora10 scripts can be run more than once in a session and the variables PATH and LD_LIBRARY_PATH to not grow to big by adding themselves to their new definition
WLCG Service Reliability Workshop
83
Host Housekeeping • Summary of environment files [oracle@lcgdb01 ~]$ l ora* -rwxrwxrwx 1 oinstall 43 -rwxrwxrwx 1 oinstall 302 -rwxrwxrwx 1 oinstall 5114 -rwxrwxrwx 1 oinstall 552 -rwxrwxrwx 1 oinstall 55 -rwxrwxrwx 1 oinstall i t ll 298
May May Mar May Aug Sep S
18 2006 ora10asm.sh 22 2006 ora10agent_cern.sh 29 2007 orahousekeep.sh p 22 2007 ora10common.sh 8 10:00 ora10ogma.sh 12 15:32 15 32 ora10agent_cclrc.sh 10 t l h
WLCG Service Reliability Workshop
84
Host Housekeeping • ora10common.sh # clusterware export CRS_HOME=/opt/oracle/crs/oracle/product/10/app # oracle export ORACLE_BASE=/opt/oracle/app/oracle export ORACLE_HOME=$ORACLE_BASE/product/10.2.0 export PATH $PATH $ORACLE HOME/bin $ORACLE HOME/OPatch $ORACLE HOME/jdk/bi PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE_HOME/jdk/bi n:$CRS_HOME/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib # sqlplus export EDITOR=vi export ORA_NLS10=$ORACLE_HOME/nls/data export NLS_LANG="english_united NLS LANG "english united kingdom kingdom.AL32UTF8" AL32UTF8" export NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS" export RMAN="$ORACLE_HOME/bin/rman target / catalog rman/@rcat" WLCG Service Reliability Workshop
85
Host Housekeeping • ora10ogma.sh /home/oracle/ora10common.sh export ORACLE_SID=ogma1
WLCG Service Reliability Workshop
86
Host Housekeeping • ora10asm.sh /home/oracle/ora10common.sh export ORACLE_SID=+ASM1
WLCG Service Reliability Workshop
87
Host Housekeeping • Agent environment script: ora10agent_cclrc.sh export ORACLE_BASE=/opt/oracle/app/oracle export ORACLE ORACLE_HOME=$ORACLE_BASE/agent10g HOME=$ORACLE BASE/agent10g export AGENT_HOME=$ORACLE_BASE/agent10g unset ORACLE_SID export PATH=$PATH:$ORACLE_HOME/lcgdb01.gridpp.rl.ac.uk/bin:$OR ACLE_HOME/OPatch:$ORACLE_HOME/jdk/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
WLCG Service Reliability Workshop
88
Host Housekeeping • Agent environment script: ora10agent_cern.sh export ORACLE_BASE=/opt/oracle/app/oracle export ORACLE ORACLE_HOME=$ORACLE_BASE/agent10g_CERN/agent10g HOME=$ORACLE BASE/agent10g CERN/agent10g export AGENT_HOME=$ORACLE_BASE/agent10g_CERN/agent10g unset ORACLE_SID export PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$ORACLE _HOME/jdk/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib
WLCG Service Reliability Workshop
89
Host Housekeeping • Agent environment scripts – Check you are using the right emctl – which hi h emctl l
• Check status with emctl status agent g • Check files are uploading • Automatically start on host boot
WLCG Service Reliability Workshop
90
Documentation • System documentation – versions, hosts, databases, users, patch versions
• wiki – Who can access?
• Disaster Recovery • Contacts • Standard Operating Procedures – startups, shutdown, user alerts
• Do you know who to ring/email in an emergency? • Do you know what to without power/telephones? WLCG Service Reliability Workshop
91
Grid Control
WLCG Service Reliability Workshop
92
WLCG Service Reliability Workshop
93
Grid Control • Secure Sec re agent upload pload • The agents as to be secured against the OMS before they can be upgrade via grid control. – ./emctl secure agent
• To secure the agent you need a registration password from the OMS. Which you can obtain or add one to use for a one off registration • Go to Grid Vontrol setup page and then click click-on on registration password link.
WLCG Service Reliability Workshop
94
Always set Target Properties
target properties
WLCG Service Reliability Workshop
95
Target Properties found in Target Home
WLCG Service Reliability Workshop
96
There is workaround for missing button!
WLCG Service Reliability Workshop
97
Check alerts - Critical
WLCG Service Reliability Workshop
98
Check alerts - Warning
WLCG Service Reliability Workshop
99
Check alerts - Errors
WLCG Service Reliability Workshop
100
Set Credentials
WLCG Service Reliability Workshop
101
Setup Email Alerts
email alerts
WLCG Service Reliability Workshop
102
Setup Templates
WLCG Service Reliability Workshop
103
Setup Metric Thresholds
WLCG Service Reliability Workshop
104
Backups
WLCG Service Reliability Workshop
105
Backups
• • • • •
Backup details stored centrally Common scripts One connection from all machines Version has to be newest Set p at RAL: Setup RAL – Dedicated server – Cold backups
WLCG Service Reliability Workshop
106
Backups – RMAN Catalog • Jobs in crontab on each database # RMAN 30 1 * 30 1 * 30 3 *
Schedules follow here * 1 /home/oracle/rmanfulback.sh mars rman rcat >/dev/null 2>&1 * 2-7 /home/oracle/rmanincback.sh mars rman rcat >/dev/null 2>&1 * * /home/oracle/rmanvalidate.sh mars rman rcat >/dev/null 2>&1
WLCG Service Reliability Workshop
107
Backups – Full Backup Script (1) #!/bin/ksh # #################################################################### # # THE INPUT PARAMETERS ARE : # # $1 - Target DB # $2 - RMAN owner # $3 - RMAN password # $4 - RMAN DB # # Script to backup archived redo log files # #################################################################### # # Check arguments # if [[ $# -ne ne 4 ]] then echo "$0 Error on $HOST: \nUsage: $0 TARGET_DB RMAN_OWNER RMAN_PWD RMAN_DB" exit 1 fi WLCG Service Reliability Workshop
108
Backups – Full Backup Script (2) ################### # RMAN variables # ################### # export SID=$1 export RMAN_OWNER=$2 export RMAN_PWD=$3 export t RMAN_DB=$4 RMAN DB $4 export LOGIN_PATH=$PATH export LOGIN_LD_LIBRARY_PATH=$LD_LIBRARY_PATH . /home/oracle/ora10$SID.sh export HBlog=/home/oracle/$SID/rmanlogs/fulback${SID}.`date +%d%m%y%H%M%S`.log echo $HBlog export emailsub=$HOSTNAME.$ORACLE_SID..Backup.Failure
WLCG Service Reliability Workshop
109
Backups – Full Backup Script (3) rman target / catalog ${RMAN_OWNER}/${RMAN_PWD}@${RMAN_DB} log \"${HBlog}\" << SQL show all; run { backup incremental level 0 database plus archivelog delete input; } report obsolete; delete noprompt obsolete; list backup; Exit SQL oraerror=`cat $HBlog | grep "ORA-"` rmanerror=`cat $HBlog | grep "RMAN-"` echo $oraerror echo $rmanerror if [ -z "$oraerror" -a -z "$rmanerror" ] then echo "No No Errors Errors" else echo "Errors Found !!!!" cat $HBlog |mailx -s $emailsub [email protected] fi WLCG Service Reliability Workshop
110
Backups – Level 1 Backup Script show all; run { backup incremental level 1 database plus archivelog delete input; } report obsolete; delete noprompt obsolete; list backup; exit
WLCG Service Reliability Workshop
111
Backups – Archive Logs Backup Script show all; run { backup archivelog all delete input; } report obsolete; delete noprompt obsolete; list backup; exit
WLCG Service Reliability Workshop
112
Backups – RMAN Sync Script • Run through OEM
show all; run { resync catalog; } list backup; exit
WLCG Service Reliability Workshop
113
Backups – Restore Validate Script
show all; run { restore validate database archivelog all; } list backup; exit
WLCG Service Reliability Workshop
114
Backup Job in Grid Control
WLCG Service Reliability Workshop
115
Backups – Backup Report by Email (1) Latest Database Backup Times Database -----------------------ATLASDLF ATLSTAGE CASTORP CASTORT CMSDLF . . MINERVA STAGERDB VENUS WARHORSE WARHORSE
DBID ---------4091293823 819435209 2263675954 2415163970 1227029095
Latest Backup Bytes Processed Seconds Taken ----------------------------- --------------- ------------26-NOV-2007 23:44:39 6021971968 3850 27-NOV-2007 00:54:13 85770240 56 27-NOV-2007 00:31:40 12722176 26 16-NOV-2007 00:31:39 23625728 28 26-NOV-2007 21:11:42 3530555392 2098
Backup Type --------------------DB INCR DB INCR DB INCR DB INCR DB INCR
402814294 1089287619 2579470848 780645177 792067278
26-NOV-2007 26-NOV-2007 26-NOV-2007 26-NOV-2007 27 27-NOV-2007 NOV 2007
DB DB DB DB DB
22:24:11 22:31:57 22:17:14 18:47:06 00:46:17
6494879744 164298752 3631218688 127975424 7471104
4843 91 2679 82 9
INCR INCR INCR INCR INCR
23 rows selected. --------------------------------------------------------------------------------------------------------------------The Following Databases have not been backed up in the last 24 hours DB_NAME -----------------------CASTORT
WLCG Service Reliability Workshop
116
Backups – Backup Report by Email (2) Latest Restore Validate Runs Database -----------------------VENUS ATLASDLF LHCBSTAG MERCURY . . WARHORSE SETH CASTORP HEKATE CMSDLF CASTORT MARS LHCBDLF
DBID ---------2579470848 4091293823 3505332007 344759533
Latest Restore Validate ----------------------------27-NOV-2007 01:11:22 27-NOV-2007 04:38:24 27-NOV-2007 02:50:41 27-NOV-2007 06:42:30
792067278 2382003523 2263675954 3791789259 1227029095 2418831922 1068316094 2704015590
27-NOV-2007 27-NOV-2007 27-NOV-2007 27-NOV-2007 27-NOV-2007 27-NOV-2007 27 NOV 2007 27-NOV-2007 27-NOV-2007
02:46:54 02:07:17 03:32:05 00:04:33 00:19:53 02:34:26 04:48:05 03:06:59
24 rows selected. -------------------------------------------------------------------------------------------------------The Following Databases have not had a Validate in the last 24 hours no rows selected
WLCG Service Reliability Workshop
117
Backups – Email Script #################################################################### # # This script Produces Backup Reports to be sent to the DBAs # # Amendments : # AJW - 15/05/2007 initial construction # #################################################################### # set the rcat environment . /home/oracle/ora10rcat.sh sqlplus / as sysdba <<-!! @/home/oracle/rmanrep1.sql exit !! export emailsub emailsub=Backup.Report. Backup Report `date date +%d%m%y%H%M%S +%d%m%y%H%M%S` echo $emailsub cat /home/oracle/rmanrep1.txt |mailx -s $emailsub [email protected]
WLCG Service Reliability Workshop
118
Backups – Email SQL spool /home/oracle/rmanrep1.txt; set line 120 set pagesize 999 /* get list of latest database backup times */ select a.db_name "Database", db.dbid "DBID", a.end_time "Latest Backup", a.output_bytes "Bytes Processed", (end_time - start_time) * 60 * 60 * 24 "Seconds Taken" from rman.rc_rman_status a, rman.rc_database db where object_type in ('DB FULL','DB INCR') and status = 'COMPLETED' and operation = 'BACKUP' and end_time = (select max(end_time) from rman.rc_rman_status b where b.db_name = a.db_name and b.db_key = a.db_key j _type yp in ('DB ( FULL','DB , INCR') ) and object and status = 'COMPLETED' and operation = 'BACKUP') and db.db_key = a.db_key order by end_time ;
WLCG Service Reliability Workshop
119
Backups – Catalog Backup Script ${ORACLE_HOME}/bin/rman target / nocatalog log \"${HBlog}\" << SQL show all; shutdown immediate startup mount; b k backup database; d t b startup; report obsolete; d l delete noprompt obsolete; b l list backup; exit
WLCG Service Reliability Workshop
120
Backups – RMAN Configuration RMAN> show all; RMAN configuration parameters are: CONFIGURE RETENTION POLICY TO REDUNDANCY 2; CONFIGURE BACKUP OPTIMIZATION ON; CONFIGURE DEFAULT DEVICE TYPE TO DISK; CONFIGURE CONTROLFILE AUTOBACKUP ON; CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO p p '/opt/oracle/backup/mars/%F.bak'; CONFIGURE DEVICE TYPE DISK BACKUP TYPE TO COMPRESSED BACKUPSET PARALLELISM 1; CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; CONFIGURE CHANNEL DEVICE TYPE DISK MAXPIECESIZE 100 M FORMAT '/opt/oracle/backup/mars/mars_%U.bak'; CONFIGURE MAXSETSIZE TO UNLIMITED; CONFIGURE ENCRYPTION FOR DATABASE OFF; CONFIGURE ENCRYPTION ALGORITHM 'AES128'; CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/opt/oracle/app/oracle/product/10.2.0/dbs/snapcf_mars1.f';
WLCG Service Reliability Workshop
121
Backups - Summary • We run backup jobs through crontab • Resync and Email job through OEM • Archive c e Logs ogs bac backup up job cou could d be run u auto automatically at ca y as oe oem job when area 70% full (for example) • Keep K it i simple, i l k keep iit d documented d
WLCG Service Reliability Workshop
122
Questions & (hopefully) Answers [email protected]
WLCG Service Reliability Workshop
123
