Transcript
Video over IP JongWon Kim, Ph.D.
KRNET 2003 June 26th, 2003 Networked Media Laboratory Dept. of Information & Communications Kwang-Ju Institute of Science & Technology (K-JIST)
[email protected]
http://netmedia.kjist.ac.kr NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Contents Media Delivery over IP
Network-adaptive Media Delivery o Networking Support o Network Adaptation
Video over IP as VoIP extension
Future of Video over IP - Toward Access Grid & Immersive Media
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Multimedia Delivery over IP
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Multimedia over IP Networks Media Broadcasting Hosted Streaming Interactive Conferencing Conferencing Proxy
Streaming Sender
RTN Proxy
Internet
Manager Receivers NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
IP (Internet Protocol) Networks IP uses packet switching o Suitable for unexpected burst of data without establishing an explicit connection; Bandwidth is IP protocol version number shared statistically header length
IP is neither reliable nor delaybounded – currently only best effort
(bytes) “type” of data
max number remaining hops (decremented at each router)
o Network congestion and failures can cause temporary packet losses. upper layer protocol o Queuing delay, especially when to deliver payload to congested Time critical applications cannot operate well o Fluctuations in available bandwidth, end-to-end delay, and loss.
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
IPv4 (RFC 791) 32 bits ver head. type of len service
length fragment 16-bit identifier flgs offset time to upper Internet layer live checksum
total datagram length (bytes) < 64k bytes for fragmentation/ reassembly
32 bit source IP address 32 bit destination IP address Options (if any)
data (variable length, typically a TCP or UDP segment)
E.g. timestamp, record route taken, pecify list of routers to visit.
Internet Network Layer & Routing Intra-AS border (exterior gateway) routers
Transport layer: TCP, UDP
Network layer
IP protocol •addressing conventions •datagram format •packet handling conventions
Routing protocols •path selection •RIP, OSPF, BGP
routing table
C.b
a
ICMP protocol •error reporting •router “signaling”
C
Host h1
A.a
b
Inter-AS routing between B.a A and B A.c
d A
a
a c
b
Intra-AS routing within AS A
Link layer
c
b B Intra-AS routing within AS B
Inter-AS interior (gateway) routers
physical layer
Router Configuration and Control Queue
Meter
Queue Multi-Field Classifier
Marker
Policer/ Shaper
Behavioral Classifier
Queue Manager
Scheduler
Queue
Traffic Conditioning NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Host h2
Per Hop Behavior
Next Generation Internet End-to-end argument: IP’s fundamental design principle is “putting smarts in the ends of the network, leaving the network core dump”
Today’s Internet
Intelligent Internet
Connectivity
New differentiated services
Commodity: “plain old data service”
Services tailored to market segmentation and value
No integrated infrastructure for service creation : Best-effort
Rapid deployment for services evolution and creation
Broadband, QoS, Multicast, Security, Convergence NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Multimedia Delivery - Technologies Efficient Media Representation Techniques: Scalable source representation to deal with dynamically varying bandwidth Robust source representation to deal with high error/loss rate
Dynamic and Reliable Media Delivery Techniques: Network adaptation to map scalable and robust source represent ations onto the MM characteristic-aware transport protocols Adaptive resource provisioning/management to guarantee the de sired QoS / multicast service
Universal Media Access with Security Techniques: Unified integrated media support with a scalable information str ucture and a choice of efficient file format for MM End-system and application customization with security concerns NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Multimedia Application’s QoS Requirements Interactive
Responsive
Timely
Non-critical
Packet Loss
5%
Conversational voice and video 0%
100 msec
Zero loss
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Comman d/ control (eg Telnet, Interactive games)
Voice/video messaging 1 sec Transactions
(eg E-commerce, Web-browsing, Email access)
Streaming audio/video 10 sec Paging, Download s (eg FTP, still image)
Delay Fax
100 sec
Background (eg Usenet)
Media Delivery – Application vs Network Network Characteristics and Policies: Rate (CBR, VBR, ABR, …), Error (random/bursty in BER/PLR/CRL), Delay/Jitter (tolerance)
Content-aware Transport
Application/Source Application/Source Characteristic : steady/burst, real-time?, lossresilient?
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Network Network-adaptive Delivery
Network Adaptive Media Delivery - Networking Support Part -
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Media Delivery – Adaptation & Support
Congestion (Rate, Flow) Control
Media Representation (Compression)
Media Synchronizati on
System Support (Transport, OS, Storage)
Dynamic Network Adaptation Framework
Error Control Network QoS Support NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Interoperable Protocol Other Middleware Support Network Multicast Support
Inter-operable Delivery with Protocol Support Media Contents
Shared Tools RTP Payload
RSVP Security Protocols
RTCP
Conference Control
SDP SAP
SIP
RTSP
HTTP
RTP UDP
TCP
IPv4, IPv6
Standard-based vs. Proprietary Protocols? Related issue: Standardized media content representation (e.g. MPEG-X) vs. Others? NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
SMTP
Media Streaming Related Protocols
Hellix
Darwin
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
MPEG-4 Example Movie window
Volume control Movie Button
Play, Pause button
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Time control
Required Network Infrastructure: Multicast, QoS, and others Paid services with Copyright management, authentication, billing, etc
Content broadcast network
End-to-end QoS guarantee with QoS provisioning (admission control and QoS controls) & Network adaptation
Economical and reliable network with
The Internet
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Scalable multicasting & Optimized server location and load sharing including contents routing
Multicast Support for Multipoint Media Distribution Multicast streaming? Model (ASM, SSM, ALM/Overlay/P2P) Source filtering capability
Internet
232/8
IGMPv3
Reliable Multicast Transport Mobile/Wireless Multicast Resilient Multicasting of continuous media NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
QoS-network Support for Quality Media Delivery QoS Model: IntServ, DiffServ, Traffic Engineering (MPLS), … Configuration Reservation request QoS monitor
TCP/IP
Console or Client User
Resource Manager-aware Application
CAC module
Execution module
Monitoring module
SNMP module
Route
Re source
Mon.
KOREN21 K-JIST
Reservation and Allocation NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
SLA module
Route Info module
DB Interface
JAVA Interface Program
DB SLA
TCP/IP
Browsing
QoS signaling
Auth. module
BBTP
Resource Manager
EF
MIB
BE
WFQ
DiffServ Network MIB
MIB
KISTI
Toward Consolidated Support via Middleware? Issues to be covered +++ o o o o
Identification Directories, … Security: Authentication, Authorization, … …
Middleware:
o Specialized networked services that are shared by applications and users o A set of core software components that permit scaling of applications and networks
GRID Middleware Home Networking Middleware NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Network Adaptive Media Delivery - Network Adaptation Part: Bandwidth (rate), loss, and delay -
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Media Delivery – Adaptation & Support
Congestion (Rate, Flow) Control
Media Representation (Compression)
Media Synchronizati on
System Support (Transport, OS, Storage)
Dynamic Network Adaptation Framework
Error Control Network QoS Support NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Interoperable Protocol Other Middleware Support Network Multicast Support
Required Network Adaptation Functionalities In case of Reliable Unicast with feedback o Network Congestion Control => Quality Adaptation: Source Rate Control / Network Rate Adaptation o Error control: Error Resiliency and Error Recovery o Quality Recovery: Post-processing & Concealment
o Synchronization Control & Adaptive Playout Quality Adaptation
Error Control Quality Recovery
Adaptable Source
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Media Pump
Congestion Control (TCP-friendly)
Network
Client
Synchronization Control
TCP-friendly Internet Video Delivery Error detection & recovery Error resilient coding
Motion Compensated Frame Interpolation Deblocking/ Deringing Filtering
Frame Rate Control RCCM for Bandwidth / Error Control
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
TCP-Friendly Congestion Control / Rate Control TCP-Friendly Congestion Control: equationbased … Toward unified congestion control handling: Congestion Manager (CM) Congestion Control / Rate Control / Flow Control Multicast congestion control? NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
TCP-Friendly Congestion Control / Rate Control (Cont.) Rate Control by o Source Rate Control (on-line encoding only) o Source Rate Adaptation (or Shaping) o Smoothing – via Buffering o Packet Scheduling o Pre-fetching
I
Transmission Rate
B
B B P
(4)
time New connection request I-picture Starting point
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
I BB P BBP BB P BB I BB P
B
B Time
Ravg
0 (3)
B
Transmission Rate
period
(2)
B
Td 2Td 3Td 4Td 5Td 6Td 7Td 8Td 9Td
Superposed stream (existing sources) (1)
B
Non-Smoothed VBR stream
P
P
BB P
B B
Smoothed near-CBR stream
I
Td 2Td 3Td 4Td 5Td 6Td 7Td 8Td 9Td
Time
Proactive and Reactive Error Controls Network error controls o Proactive (FEC), Reactive (ARQ), and Hybrid o Delay is key controlling factor
Network (Channel) error control vs Source error resiliency?
Quality Recovery: Error Concealment and Postprocessing
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Error Control and Quality Recovery Source
Network Adaptation
Encoder
Network Adaptation Decoder
CRC
Sink
Error Detection
Sequence No.
Loss Detection
Retransmission Error Correction
FEC Data Structuring (Synchronization)
Erasure Correction Resynchronization
Quality Control (Layered Coding)
Loss Concealment Network Framing
Post Processing
Application Framing Coding NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Courtesy of Dr. Noh NVW 2001
Synchronization Control and Adaptive Playout
Synchronization o Intra-media o Inter-media o Inter-client
o Tight vs Loosely-coupled
Adaptive playout o Utilizing silence o Utilizing time-scale modification NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Inter-relationship of QoS Factors Network Packet Loss
Overall Packet Loss Codec Performance
Network Jitter
Perceived Quality
Jitter Buffers Overall Delay
Network Delay
Network Factors NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Application Factors
QoS Service Level
Media Delivery – Network Adaptation MM Content
Source encoding Feedback from Host/Application
Scalable stream in rate/error Source transcoding & Framing/Multiplexing Integrated &
Feedback from Network
Channel stream
prioritized stream Trans-MUX & Channel coding with FEC/ARQ
Network Transmission
< Source prioritization for content-aware network adaptation > NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Media Delivery – Network Adaptation Layered Video Encoding
Network Adaptation & Prioritized Packetization
R-D / Corruption Model
R-D Analysis
Video Layered Preenprocesscoding Corruping tion Analysis
Frame Complexity (Quality Constraint)
Constant Quality Rate Control
Target Minimum Bandwidth (Network Constraint) Target Minimum Buffer Size (Receiver Constraint)
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Network Adaptation
Layered RPI
Wired/Wireless Networks
Network Adaptation
(Source Prioritized Rate/Error PacketiResilience) zation (Network Rate/Loss/ (only if Delay) applicable)
Estimated Available Bandwidth, Loss/Delay
Network Adaptation Network (Network Adaptation Filtering)
Network Feedback
Network Monitoring & Feedback Handling
Receiver/ User
(Receiver / User Adaptation)
Network Feedback (end-to-end) Application Feedback
Video over IP as VoIP extension (V2oIP)
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
VoIP Extension: Voice + Video + Data We want to achieve IP TELEPHONY (a.k.a. voice/video/data convergence): Real time voice, video and data communication over Internet Protocol (IP) networks However, latest high-end VC system supports o o o o o o o o
H.263+ or H.263++ (w/ Annex U for background restoration) 384 kbit/s or 768 kbit/s TDM/ISDN (H.320) or IP (H.323) CIF (352x288) or CIF-Interlace 2x(352x288) Capture and rendering typically interlace 25/30 fps, 50/60 fields per second Round-trip delay around 400ms + Round-trip delay for multipoint (w/ MCU) more than one second
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
V2oIP Building Blocks
Protocol stacks (H.323, MGCP, SIP) Switching cores (e.g. Gatekeepers, Call Agents) Media engines (Announcement, Mixing, Gateway) Service creation environment
Call Agents
Proxy
SIP
Service Service Creation Announcement Conference Bridges Creation Player
Announcement Toolkit
Multi-point Control
MGCP
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
GateGatekeepers keepers
Gatekeeper Toolkit
H.323
Gateways Gateways
V2oIP Deployment
MCU
POTS
PSTN Multicast Audio and Video Decentralized
Unicast Audio and Video Centralized H.323 Entity TC
Non-H.323 Entity TP
xDSL Non-H.323 System
Administrative Domain A
Physical Connection
GKA1
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
BECH
GKC/ BEC
GW
Signaling
H.323 Entity TA
Clearing House
GKC1
GKA/BEA
Cable BEB/GKB
Administrative Domain A
GKB1
Administrative Domain B
Backend Servers (services)
H.323 Entity TB
Video Quality and Function Issues (1) Improve Video Quality and Size Upcoming H.264 video encoding o o o
Product design stage now, 640x480 at full frame rate on one high-end DSP Bit rate around 384 kbit/s yields acceptable picture quality Subjective quality significantly better than H.263++ w/ CIF at same bit rate
o
Linking with region-oriented (maybe combined with segmentation) encoding on top of variable frame skipping
Variable, content-dependent capture rate
Wide-screen video support o
Can support video resolution like 2560x480 with single CCD camera with custom, wide-angle lens; Multiple DSPs for distributed codec is required; 1.5 – 3 Mbps
Progressive scan hardware o
No A/D and D/A losses; Better quality and good delay characteristics; 250ms round-trip delay over IP demonstrated?
Enhance error resilience with ERPS and so on NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Video Quality and Function Issues (2) Provide more natural, cooperative environment for multi-site conferencing Let’s avoid transcoding and enhance user experience
o Create all representations needed in the sending endpoint and simulcast them (in line with layered multicast video) o Ideally suited for IP multicast environment; But can be with overlay (application-layer) multicast; o May need some innovative changes in the control and mux protocols o No transcoding artifacts whatsoever o No additional end-to-end delay for multipoint when using IP multicast
Let’s get rid of fixed environment that discourages uncooperative use of the technology
o Toward flexible meeting environment without fixed camera position, fixed background, studio lighting, seating
Let’s add more functionalities while keeping user interface simple and straightforward NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
VoIP Extension: Real-time Collaboration Bring Voice, Screen, Computation together to improve: Realtime Collaboration Vision from Microsoft o o o o o
Rich presence and IM PC and phone integration Data/screen conferencing Information agent Enhanced Meetings incoming communi cation
context
to web phone to task list
Automatic Broadcast and Recording Remote Participation NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Note taking 2003 Gurdeep Singh Pall from Microsoft Corporation
Future of Video over IP - Access Grid & Immersive Media -
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
The Access Grid Access Grid does for people what the computational Grid does for machines Much more than teleconferencing, if possible at commodity prices To enable groups of people to interact with Grid resources and to use the Grid technology to support group to group collaboration at a distance o o o o
Distributed Lectures and seminars Remote participation in panel discussions Virtual site visits meetings Complex distributed grid based demonstrations
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Usage Areas The Academic and Research, Government, Private Sectors Making Remote Collaborations Work across Boundaries Check “Multi-Sector Collaboration over the Access Grid” by J. T. von Hoffman (Boston Univ)”
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Access Grid Components of an AG Node RGB Video
Digital Video
Display Computer
Digital Video
NETWORK
Digital Audio
Video Capture Computer
NTSC Video
Audio Capture Computer
Analog Audio
Mixer Control Computer
Presenter mic
Echo Canceller
Stream Type
Max. latency
Max. Jitter
Min. BW
Max. Loss
Multi cast
Text
100 ms
N/A
64 kbps
0%
No
Audio
400 ms
60 ms
64 kbps
5%
Yes
256 kbps x 4 = 1 Mbps
25 %
Video
400 ms
30 ms
Yes
Presenter camera Ambient mic (tabletop) Audience camera
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Requires Multicast
AG extended toward ACE
Enable group-to-group interactions at a distance Improve the user experience Enable complex multi-site visual and collaborative experiences Build on integrated grid services architecture Use quality but affordable digital IP based audio/video
Immersive Media Supported Advanced Collaboration Environment Advanced Collaboration Environment
Access Grid 2.0
Access Grid NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Broadband Content Requires More… Browsing 144 Kbps
½ Screen Video 300Kbps
Music CDs 160 Kbps
Full Screen 600Kbps – 1.5Mbps
TV Shows
Music
High Resolution 6-8 Mbps
Live Events
Pay Per View
What kinds of broadband contents o o o o
High speed Internet access: WWW + Mail + Messaging Video focused: Basic TV + Pay TV + Personalized TV Networked games + Music downloads + Online gambling …
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Immersive Media > 100 Mbps
Future Broadband Contents Service & Immersive Media Interactive collaboration - Collaborative design and engineering; Remote customer support; Distance learning Remote visualization & Large-scale, multi-site computation and data mining Shared virtual reality - Military and industrial team training / simulation; Multiplayer games; Virtual shopping malls (e-commerce); Mobile entertainment; Online tradeshows and conferences; Virtual Heritage Any combination of the above
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Core Techniques for Immersive Media Service Integration 3D Video Synthesis & Reconstruction
Immersive Media Storage & Query/Browsing
3D Geometry Generation & Modeling
Immersive Media Content Adaptation
Interactive User Interface
Immersive Media Compression & Integration
Immersive Media Delivery
Contents Service (Protocol & System)
3D Audio Generation & Reconstruction
Multi-modal Immersive Media Service Technology Multi-D Haptic Generation & Reconstruction
NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
Digital Contents Management & Protection
Thank You! Send Inquiries to
[email protected] Access Grid: o http://accessgrid.org o http://www.accessgrid.or.kr
Advanced Networking in Korea o http://anf.ne.kr NETWORKed MEDIA LAB. DEPT. OF INFO. & COMM., K-JIST
