Transcript
CON6761 Online Database Maintenance with No Impact to Applica:ons Carol Colrain
Technical Lead for Client-Failover, RAC Development
Troy Anthony
Cloud Evangelist, RAC Development
Takashi Ito
Sr. SoLware Engineer, NEC CorporaOon
Michael Timpanaro-PerroRa
Senior Director Product Management, RAC Development
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement The following is intended to outline our general product direcOon. It is intended for informaOon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcOonality, and should not be relied upon in making purchasing decisions. The development, release, and Oming of any features or funcOonality described for Oracle’s products remains at the sole discreOon of Oracle.
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
3
Announcing Oracle Database 12c Release 2 on Oracle Cloud • Available now – Exadata Express Cloud Service
• Coming soon – Database Cloud Services – Exadata Cloud Machine
Oracle is presenOng features for Oracle Database 12c Release 2 on Oracle Cloud. We will announce availability of the On-Prem release someOme aLer Open World. Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
4
Program Agenda 1
How do we schedule server maintenance?
2
Are the building blocks in place ?
3
Draining SoluOons
4
Older Style ApplicaOons
5
Set and Forget
6
NEC Success Story
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | ConfidenOal – Oracle Internal/Restricted/Highly Restricted
5
1
How do we schedule server maintenance ?
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
6
Users should see no errors during maintenance Preventable Situa:on There is no reason for users to see downOme during scheduled database maintenance – Service is unavailable – ApplicaOon owners unable to agree maintenance windows – Long running jobs see errors – DBA’s and engineers work off hours – ApplicaOon and middleware components need to be restarted
Family Holidays website is down "Our online transacOon services are currently unavailable. Our server may be temporarily down or we may be performing rouOne maintenance funcOons scheduled every Sunday from 12 a.m. to 5 a.m. (Eastern Standard Time). We apologize for any inconvenience."
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
7
How do we solve for all your applicaOons? Move work to different instance/database with no errors reported to applicaOons Transparent to applicaOons and mid-Oers Support all server side maintenance patches, PSUs, repairs, changes,
major release, unplug/plug, migraOon, expansion, h/w replacement
Configure once, same for all commands
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
8
High Availability by Patch Type One- Off
PSU/CPU
Bundle Patch
Patch Set
96%
All
Most
No
98%
All
All
No
Online - Hot
82%*
No
No
No
GoldenGate
All
All
All
All
RAC Rolling DG Standby First
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | Oracle ConfidenOal – Internal/Restricted/Highly Restricted
9
2
Are your building blocks in place ?
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
10
Affinity and Draining Locally - Switchover across Sites ProducOon Site
Fast ApplicaOon NoOficaOon ApplicaOon ConOnuity, TransacOon Guard – Applica3on HA
AcOve Data Guard
Global Data Services
RAC
– Online Rolling Maintenance – Scalability – Server HA
– Scheduled switchover – Data Protec3on, DR – Query Offload
– Cross Site Placement
Data Guard
RAC One
– Online Rolling Maintenance – Server HA
Replicas
– No3fy draining, failover, load balancing
– Scheduled switchover – Data Protec3on, DR
Drain within RAC
Drain within RAC Switchover ADG or GG
GoldenGate
– Scheduled switchover – Ac3ve-ac3ve replica3on – Heterogeneous
Sharding
– Massive OLTP – Scheduled switchover – Ac3ve-ac3ve replica3on – Heterogeneous
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
11
Oracle RAC One Node and Oracle RAC Drain services to another ac:ve instance gradually
Oracle RAC: mul:ple instances per database running concurrently
Oracle RAC One Node: one instance per database running at a Ome, both instances during maintenance “Always On”
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
“Always Running”
12
12cR2 Oracle Ac:ve Data Guard Con:nuous Service Availability
Drain primary and switch Switchover to
Data Guard Broker
[wait]
ADG sessions survive standby role change ApplicaOon ConOnuity recovers stragglers
SWITCHOVER
Site A
RAC Primary
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Site B
RAC Standby
13
12cR2 Oracle Mul:tenant Con:nuous Service Availability
TNS COMMON LISTENER REGISTRATION
TNS LISTENER
Online PDB relocate Drain and cut over in under 10 seconds
GL
AP
CDB1
PO
OE
OE OE Relocate and Drain Read service ahead
CDB2
Final REDO Apply Applica:on Con:nuity moves stragglers Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
14
Use Services for Loca:on Transparency Services provide a “dial in number” for your applicaOon
• Regardless of locaOon, applicaOon keeps the name • Moving, reshaping, prioriOzing controls how a service is offered • Batch and OLTP separated RAC instance
RAC instance
Node 1
Node 2
• DB and PDB names for admin only OLTP service Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Batch service 15
Are connec:ons properly configured? Automa:c Retries
alias =(DESCRIPTION = (CONNECT_TIMEOUT=90) (RETRY_COUNT=20)(RETRY_DELAY=3) (TRANSPORT_CONNECT_TIMEOUT=3) (ADDRESS_LIST = (LOAD_BALANCE=on) ( ADDRESS = (PROTOCOL = TCP)(HOST=primary-scan)(PORT=1521))) (ADDRESS_LIST = No reliance on DNS (LOAD_BALANCE=on) Configure in One Place ( ADDRESS = (PROTOCOL = TCP)(HOST=secondary-scan)(PORT=1521))) LDAP or TNS names (CONNECT_DATA=(SERVICE_NAME = gold-cloud)))
ALWAYS use a service that is NOT DB/PDB name Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
16
Align the Timeouts NO interruption when Active/ Active RAC
Application Timeout Ø Drain + Switch to DG DG Broker
Switchover
Stop or Relocate Service Drain Work
Drain Work RAC Primary
RAC Standby Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Switchover 17
3
You are now ready to drain the work
Applica:ons should see no errors during maintenance.
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
18
Roll Maintenance - Drain work at safe places Move services in advance Replace connec:ons where applica:ons don’t no:ce Ø ConnecOon pools Ø ConnecOon tests Ø TransacOon boundaries
ConOnue on a new connecOon with states restored
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
19
DBA steps – Works Today Repeat for each service allowing Ome to drain • Stop service (no –force) srvctl stop service –db .. -instance .. -service .. (omit service for all)
• or Relocate service (no –force) srvctl relocate service –db .. -service .. –oldinst .. –newinst srvctl relocate service –db .. -service .. –currentnode.. –targetnode
• Wait for sessions and transac:ons to drain • For remaining sessions, stop transac:onal exec dbms_service.disconnect_session( ‘… your service ..‘, DBMS_SERVICE.POST_TRANSACTION);
• Stop the instances using your preferred tool; op:on to disable Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
20
Draining with Oracle ConnecOon Pools – Works Today Applica:ons using …
Oracle – WebLogic Ac:ve GridLink, UCP, OCI, ODP.NET managed and unmanaged, OCI Session Pool, Tuxedo 3rd party App Servers using UCP: IBM WebSphere, Apache Tomcat, NEC WebOTX
DBA Step
srvctl [relocate|stop] service (no –force) Immediately new work is redirected
Sessions Drain
FAN Planned
Pools drain sessions as work completes
Gradually AcOve sessions are released when returned to pools
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
21
FAN with other Java-Based App Servers – Works Today Use UCP – a simple Data Source replacement
IBM WebSphere Apache Tomcat NEC WebOTX See OTN.
Pool Data Source
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
22
12c FAN: Standardized, Auto-Configured (HIDDEN) All Oracle use ONS
Auto-Configured
JDBC Universal ConnecOon Pool JDBC Thin Driver (12.2) OCI/OCCI driver ODP.NET Unmanaged Provider (OCI)
DESCRIPTION = (CONNECT_TIMEOUT=90) (RETRY_COUNT=20)(RETRY_DELAY=3) (TRANSPORT_CONNECT_TIMEOUT=3) (ADDRESS_LIST =
ONS Node Set 1
ODP.NET Managed Provider (C#)
(LOAD_BALANCE=on)
OCI Session Pool
( ADDRESS = (PROTOCOL = TCP) (HOST=primary-scan) (PORT=1521)))
WebLogic AcOve GridLink Tuxedo Listener
(ADDRESS_LIST =
ONS Node Set 2 = (PROTOCOL = TCP)
(LOAD_BALANCE=on)
( ADDRESS (HOST=second-scan)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME=gold))) Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
23
Keep it simple – Stagger draining services
Use BATCH services
Use OLTP services
Seconds Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
24
Stagger Draining – Batch then Online Services • Use at least two applicaOon services – online and batch/backend • Start draining batch early • Online drains in seconds First Instance
Online
Batch
Online
Shutdown
Quiesce Batch for X hours… Quiesce OLTP for Y seconds
Last Instance
Online
Complete Maintenance
Online
Batch
Online
Shutdown
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
Online
25
Eliminate Logon Storms 12c Drains Gradually
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
26
12cR2 FAN Use FAN to Invoke Draining Supports 11.2, 12.1, and later databases
All Applications On by Default, Auto-Configured
Drivers detect, break-out, and drain
rd Oracle & 3 Par:es Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
27
12cR2 FAN Enable Connec:on Tests to Drain Supports 11.2, 12.1, and later databases
When asked, “ its not good”` • isValid • isUsable • isClosed • pingDatabase • SQL Tests • OCI_ATTR_SERVER_STATUS Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
28
Drain Applica:on Servers at Safe Places Applica:on Server
Test Name
Connec:on Test to DB
Oracle WebLogic – Generic and Mul: data sources
TestConnec:onsOnReserve TestConnec:onsOnCreate
isUsable SQL – SELECT 1 FROM DUAL
Oracle WebLogic Ac:ve GridLink
embedded
isUsable
IBM WebSphere
PreTest Connec:ons
SQL - SELECT USER FROM DUAL
RedHat JBoss
check-valid-connec:on-sql
SQL - SELECT COUNT(*) FROM DUAL
Apache TomCat
TestonBorrow TestonRelease
SQL - SELECT 1 FROM DUAL
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
29
Drain Applica:ons at Safe Places Applica:on
Condi:on
Connec:on Test to DB TestConnec:onsOnReserve with "BEGIN NULL; END;"
eBusiness Suite
Connec:on borrowed from Weblogic
Fusion Applica:ons
TestConnec:onsOnReserve Connec:on returned to WebLogic and C++ with isValid pools and checked OCI_ATTR_SERVER_STATUS
Siebel
EAI (batch) checks status before star:ng
OCI_ATTR_SERVER_STATUS
Customer
Custom pool with Meta data table Checks status every 60s
OCI_ATTR_SERVER_STATUS
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
30
Planned Draining Demonstra:on
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
31
4
DBA Opera:on’s Simplified
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
32
12cR2 One command for maintenance opera:ons Focus on user experience
• Group opera:ons at pdb, instance, node and database for services • New service aRributes set once
drain_:meout (seconds) stopop:on (immediate, transacOonal)
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
33
5
Transac:onal Disconnect For your older style apps
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
34
Some ApplicaOons have Read-Only Failover Built-In Applica:on
Set and Forget
Siebel
TNS + STOPOPTION = TransacOonal Drain_Timeout
PeopleSou JD Edwards Informa:ca
FAN on for unplanned Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
35
12cR2 TAF SELECT with TransacOon Guard
Hides scheduled maintenance and unplanned for read-only apps that set state at start
With stopopOon transacOonal, fails over, validates with TransacOon Guard, resets common states
1. Database Request
6. Response
2. DB Calls
4. TAF: New session + Transac:on Guard + common states 5 DB calls con:nue 3. Disconnect Error 12c ORACLE Database
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
36 36
12cR2 Applica:on Con:nuity for Longer Running • Applications that borrow a connection and do not return until completed • Purges committed transactions as it goes • Stop option Transactional, Drain Timeout “n“ • Rules apply
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
37
ApplicaOon ConOnuity – Long Running
Hides scheduled maintenance and unplanned for static applications
At disconnect transacOonal, fails over, validates with TG, syncs states, replays
1. Database Request
6. Response
2. DB Calls
Appears to applicaOons 4. AC : New session + Transac:on Guard + as a delayed execuOon. Simple states + Replays 5 DB calls con:nue 3. Disconnect Error 12c ORACLE Database
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
38
6
Set and Forget
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
39
Lessons Learned • For ConOnuous Service, use an acOve/acOve configuraOon • Always use applicaOon services • Focus on the user experience • Allow applicaOon work to complete before progressing to scheduled maintenance • Drain the work with no applica:on changes • Disconnect older style applicaOons transacOonal at the service level •
12.2
Set your drain :me and stop op:on at the service level and you are done
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
40
Take Away
1. Does your environment support rolling? 2. Are connecOons properly configured? 3. Is the applicaOon well behaved? 4. What is your drain Omeout? Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
41
720 5GM H N MDHG 5TRUFS FRJ2 7 FG N MJI2 UJWF NTS 2
75 5TWUTWF NTS ) & )011 FUFSJ J R NSF NTSF UWT NIJW T NS TWRF NTS J MST TL #:C JW N J FSI UWTI
Business activities in over
140
Countries and territories Europe, Middle East, and Africa
North America
18
23
9
AsiaPacific
Latin America
19
10
25 720 0H IH MDHG
Greater China
Billion dollar in FY2014 sales
4D MH P HA 720#8 C 720 G 8
E
E EED G
EED G D
HGMDGNHN HO
N
M
HA
GMN P
2015 Zero error solution enhancement for cloud 2013 Zero error planned DB maintenance and unplanned DB outage solution 2012
ETS 9 TGF FW SJW 3 FWI2 6F FGF J () & ().
2008 2006
2005 2000 1997 1987
NEC high available Linux DB platform
NEC’s RAC 11gR1 fast failover best practice
NEC’s RAC 10gR2 fast failover best practice
STA (Strategic Technology Alliance) started
Development alliance for mission critical systems
NEC and BEA alliance started
NEC and Oracle OEM contract started(first in Japan)
720 0H IH MDHG
M 2GOD HG § § § § §
GM
WF J 64 ) A35 #BJW N J WF J N JSFS WF J EJG TLN BJW JW ) WF J 3 N J 6F F 9 FWI FOTW MT UN F FUU N F NTS
83
3UU N F NTS 5TS NS N
3 N J 9WNI NSP
EJG TLN BJW JW GridLink DS1
GridLink DS2
A35 TIJ) 64)
A35 TIJ 64
GridLink DS3
A35 TIJ+ 64+ 564
720 0H IH MDHG
GridLink DS252
A35 TIJ
A35 TIJ,
64)
64
6F F 9 FWI
64
A35 TIJ. 64+ 564
6464
M )
64 JW N J
TU
A35 STIJ RFNS JSFS J A35
A35 N
BJW N J TW 64)
M
EJG TLN 3 N J 9WNI NSP
T :RUF
N
M
BJW N J TW 64
83
N
91/
91/ 91/
91/
BJW N J TW 64)
M
EJG TLN 3 N J 9WNI NSP
T :RUF
83 N
BJW N J TW 64
M
6WFNS AJ T F J TSSJ NTS
N
0ED GM AH G GM
T :RUF D J F J
N
• •
M
M
BJW N J TW 64
01/
N JSFS
Gradual connection relocate Load Balance 720 0H IH MDHG
6WFNS AJ T F J TSSJ NTS
0ED GM AH G GM
BJW N J TW 64)
91/ 91/
Maintenance
Maintenance 0ED GM AH G GM
91/
N
0ED GM AH G GM
T :RUF D J F J
N
• • •
M
M
BJW N J TW 64)
BJW N J TW 64
01/
N JSFS
Applying Patch Set Update Configuration change Hardware maintenance
91/
M + 6F F 5JS JW
FNS JSFS J
6F F 5JS JW) EJG TLN 3 N J 9WNI NSP
T :RUF
N
M
N
M
,
91/ 91/
N
M
0ED GM AH G GM
EJG TLN 3 N J 9WNI NSP
T :RUF A35
6F F 9 FWI
01/
6WFNS AJ T F J TSSJ NTS
0ED GM AH G GM
T :RUF D J F J
•
N
M
N
M
N
M
N
M
Maintenance
91/
83 M
91/
91/ 91/
N
M
BJW N J TW 64)
N
M
BJW N J TW 64
83
0ED GM AH G GM
N
M
BJW N J TW 64)
N
M
BJW N J TW 64
A35
01/
91/
6F F 5JS JW
T :RUF D J F J
91/
N
M
N
M
BJW N J TW 64)
• • •
BJW N J TW 64)
A35
91/ 91/
91/
564)
6WFNS 8FN T JW TSSJ NTS
0ED GM AH G GM
Data Center maintenance 720 0H IH MDHG
64 AJ T F J
Maintenance
91/
N
64 RNLWF NTS G
91/
64 WJ T F J DULWFIJ 564
Migration from on-premise to cloud Major version change Data center hardware replacement
0 6NEMDM G GM 0HGR
GH D I
7H
6 DGM G G
M MH ED GM 2 WJ J FSI WJ UTS J
T :RUF DM MD
/HMC 7H
8GEP 7H
D E
MC G
/HMC 7H
EJG TLN
A35
A35
TIJ!)
TIJ!
83
Drain Session
) +
720 0H IH MDHG
B TU BJW N J
83 Maintenance B TU :S FS J
AJ FW BJW N J
1 M 0 GM 0HGR
6 DGM G G
GH D I
N DGB
0
1 M 3N
M MH ED GM 2 WJ J FSI WJ UTS J
T :RUF 83
C B TSSJ NTS WJ W
83
EJG TLN DULWFIJ
A35!)
) ( J
B FSIG
)
WNRFW
B N MT JW
A35!
WNRFW
720 0H IH MDHG
Drain Session
FN
B FW BJW N J
Maintenance
B FSIG
91/ 8GEDG
EH
0HGR
H
GH
M MH EE M G GM
# UJS B FW BJW N J
ED GM
3UU N F NTS 5TS NS N JSFG J ST JWWTW WJU F
EJG TLN
564!)
564!
MJW
A76
64 S NSJ 5TU
NLWF NTS 64 MJW 64 #) AJ T F J
-
720 0H IH MDHG
S
WJ J FSI WJ UTS J
The combinatorial solution with FAN, Application Continuity, Real Application Clusters, Data Guard, WebLogic Server Active GridLink and NEC hardware and middleware enables us to provide incredibly high available system for our Mission Critical customers. This solution will become our primary solution for cloud and big data areas.
Yuki Moriyama Senior Manager, NEC Corporation
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
50
Safe Harbor Statement The preceding is intended to outline our general product direcOon. It is intended for informaOon purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or funcOonality, and should not be relied upon in making purchasing decisions. The development, release, and Oming of any features or funcOonality described for Oracle’s products remains at the sole discreOon of Oracle.
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. |
51
Copyright © 2016, Oracle and/or its affiliates. All rights reserved. | ConfidenOal – Oracle Internal/Restricted/Highly Restricted
52
