Media Foundation

Frbiz Site


Architecture



Media Foundation Architecture mosaic pattern tile

mosaic tile backsplash

The MF architecture is divided into the Control layer, Core Layer and the Platform layer. The core layer encapsulates most of the functionality of Media Foundation. It consists of the media foundation pipeline, which has three components: Media Source, Media Sink and Media Foundation Transforms (MFT). A media source is an object that acts as the source of multimedia data, either compressed or uncompressed. It can encapsulate various data sources, like a file, or a network server or even a camcorder, with source specific functionality abstracted by a common interface. A source object can use a source resolver object which creates a media source from an URI, file or bytestream. Support for non-standard protocols can be added by creating a source resolver for them. A source object can also use a sequencer object to use a sequence of sources (a playlist) or to coalesce multiple sources into single logical source. A media sink is the recipient of processed multimedia data. A media sink can either be a renderer sink, which renders the content on an output device, or an archive sink, which saves the content onto a persistent storage system such as a file. A renderer sink takes uncompressed data as input whereas an archive sink can take either compressed or uncompressed data, depending on the output type. The data from media sources to sinks are acted upon by MFTs; MFTs are certain functions which transform the data into another form. MFTs can include multiplexers and demultiplexers, codecs or DSP effects like reverb. The core layer uses services like file access and networking and clock synchronization to time the multimedia rendering. These are part of the Platform layer, which provides services necessary for accessing the source and sink byte streams, presentation clocks and an object model that lets the core layer components function asynchronously, and is generally implemented as OS services. Pausing, stopping, fast forward, reverse or time-compression can be achieved by controlling the presentation clock.



However, the media pipeline components are not connected; rather they are just presented as discrete components. An application running in the Control layer has to choose which source types, transforms and sinks are needed for the particular video processing task at hand, and set up the "connections" between the components (a topology) to complete the data flow pipeline. For example, to play back a compressed audio/video file, the pipeline will consist of a file source object, a demultiplexer for the specific file container format to split the audio and video streams, codecs to decompress the audio and video streams, DSP processors for audio and video effects and finally the EVR renderer, in sequence. Or for a video capture application, the camcorder will act as video and audio sources, on which codec MFTs will work to compress the data and feed to a multiplexer that coalesces the streams into a container; and finally a file sink or a network sink will write it to a file or stream over a network. The application also has to co-ordinate the flow of data between the pipeline components. The control layer has to "pull" (request) samples from one pipeline component and pass it onto the next component in order to achieve data flow within the pipeline. This is in contrast to DirectShow's "push" model where a pipeline component pushes data to the next component. Media Foundation allows content protection by hosting the pipeline within a protected execution environment, called the Protected Media Path. The control layer components are required to propagate the data through the pipeline at a rate that the rendering synchronizes with the presentation clock. The rate (or time) of rendering is embedded as a part of the multimedia stream as metadata. The source objects extract the metadata and pass it over. Metadata is of two types: coded metadata, which is information about bit rate and presentation timings, and descriptive metadata, like title and author names. Coded metadata is handed over to the object that controls the pipeline session, and descriptive metadata is exposed for the application to use if it chooses to.



Media Foundation provides a Media Session object that can be used to setup the topologies, and facilitate a data flow, without the application doing it explicitly. It exists in the control layer, and exposes a Topology loader object. The application specifies the required pipeline topology to the loader, which then creates the necessary connections between the components. The media session object manages the job of synchronizing with the presentation clock. It creates the presentation clock object, and passes a reference to it to the sink. It then uses the timer events from the clock to propagate data along the pipeline. It also changes the state of the clock to handle pause, stop or resume requests from the application.



Media Foundation Transform



Media Foundation Transforms (MFTs) represent a generic model for processing media data. They are used in Media Foundation primarily to implement decoders, encoders, mixers and digital signal processors (DSPs) between media sources and media sinks. Media Foundation Transforms are an evolution of the transform model first introduced with DirectX Media Objects (DMOs). Hybrid DMO/MFT Objects can also be created. Applications can use MFTs inside the Media Foundation pipeline, or use them directly as stand-alone objects. MFTs also support hardware-accelerated video processing and their behaviors are more clearly specified. MFTs can be any of the following type:



Audio and video codecs



Audio and video effects



Multiplexers and demultiplexers



Tees



Color-space converters



Sample-rate converters



Video scalers



Microsoft recommends developers to write a Media Foundation Transform instead of a DirectShow filter, for Windows Vista & Windows 7. For video editing and video capture, Microsoft recommends using DirectShow as they are not the primary focus of Media Foundation in Windows Vista.



Enhanced Video Renderer



Media Foundation uses the Enhanced Video Renderer (EVR) for rendering video content, which acts as a mixer as well. It can mix up to 16 simultaneous streams, with the first stream being a reference stream. All but the reference stream can have per-pixel transparency information, as well as any specified z-order. The reference stream cannot have transparent pixels, and has a fixed z-order position, at the back of all streams. The final image is composited onto a single surface by coloring each pixel according to the color and transparency of the corresponding pixel in all streams.



Internally, the EVR uses a mixer object for mixing the streams. It can also deinterlace the output and apply color correction, if required. The composited frame is handed off to a presenter object, which schedules them for rendering onto a Direct3D device, which it shares with the DWM and other applications using the device. The frame rate of the output video is synchronized with the frame rate of the reference stream. If any of the other streams (called substreams) have a different frame rate, EVR discards the extra frames (if the substream has a higher frame rate), or uses the same frame more than once (if it has a slower frame rate).



Supported media formats



Windows Media Audio, Windows Media Video and MP3 are the default supported formats. Format support is extensible; developers can add support for other formats by writing decoder MFTs and/or custom Media Sources. MIDI playback is also not yet supported using Media Foundation.



Benefits over DirectShow



Media Foundation offers the following benefits:



Is scalable for high-definition content and DRM-protected content.



Allows DirectX Video Acceleration to be used outside of the DirectShow infrastructure. DXVA 2.0 is available to user-mode components without using the DirectShow video renderer.



Provides better resilience to CPU, I/O, and memory stress for low-latency glitch-free playback of audio and video. Video tearing has been minimized. The improved video processing support also enables high color spaces and enhanced full-screen playback. Enhanced video renderer (EVR) which is also available for DirectShow, offers better timing support and improved video processing.



Media Foundation extensibility enables different content protection systems to operate together.



Media Foundation uses the Multimedia Class Scheduler Service (MMCSS), a new system service in Windows Vista & Windows 7. MMCSS enables multimedia applications to ensure that their time-sensitive processing receives prioritized access to CPU resources.



Media Foundation accompanies two other technologies - Direct3D 10 and Windows Presentation Foundation - to keep pace with graphics and multimedia hardware evolution and demanding multimedia applications.



Application support



Media Foundation, for this initial release in Windows Vista and later release in Windows 7, finds use in media playback applications. Until now, mainly internal or bundled Windows services and applications are using Media Foundation.



Windows Protected Media Path (PMP), for instance, relies completely on Media Foundation.



Windows Media Player 11 in Windows Vista relies on Media Foundation for playing ASF (WMA and WMV) content and protected content, but can also use DirectShow or the Windows Media Format SDK instead. In the case of WMV9 playback, this also implies using DXVA 2.0 instead of DXVA 1.0 when the video hardware supports WMV9/VC-1 decoding acceleration.



Windows Media Player 12 in Windows 7



DirectX Video Acceleration (DXVA) 2.0, the hardware video acceleration pipeline for Windows Vista and Windows 7, also bases on Media Foundation.



References



^ Migrating from DirectShow to Media Foundation and comparison of the two



External links



Microsoft Media Foundation SDK



Media Foundation Development Forum



Media Foundation Team Blog (with samples)



Media Source Metadata



Media Foundation Pipeline



Media Foundation Architecture



About the Media Session



About the Media Foundation SDK



Enhanced Video Renderer



v  d  e



Microsoft APIs and frameworks



Graphics



Desktop Window Manager  Direct2D  Direct3D (extensions)   GDI / GDI+  WPF  Windows Color System  Windows Image Acquisition  Windows Imaging Component



Audio



DirectMusic  DirectSound  DirectX plugin  XACT  Speech API



Multimedia



DirectX  (Media Objects  Video Acceleration)  DirectInput  DirectPlay  DirectShow  Image Mastering API  Managed DirectX  Media Foundation  XNA  Windows Media



Web



MSHTML  RSS Platform  JScript  VBScript  BHO  XDR  SideBar Gadgets



Data access



Data Access Components  Extensible Storage Engine  ADO.NET  ADO.NET Entity Framework  Sync Framework  Jet Engine  MSXML  OLE DB  OPC



Networking



Winsock (LSP)  Winsock Kernel  Filtering Platform  Network Driver Interface Specification  Windows Rally  BITS  P2P API  MSMQ  MS MPI



Communication



Messaging API  Telephony API  WCF



Administration and



management



Win32 console  Windows Script Host  WMI (extensions)  PowerShell  Task Scheduler  Offline Files  Shadow Copy  Windows Installer  Error Reporting  Event Log  Common Log File System



Component model



COM  COM+  ActiveX  Distributed Component Object Model  .NET Framework



Libraries



Base Class Library (BCL)  Microsoft Foundation Classes (MFC)  Active Template Library (ATL)  Windows Template Library (WTL)



Device drivers



Windows Driver Model   Windows Driver Foundation (KMDF  UMDF)  WDDM  NDIS  UAA  Broadcast Driver Architecture   VxD



Security



Crypto API (CAPICOM)  Windows CardSpace  Data Protection API  Security Support Provider Interface (SSPI)



.NET



ASP.NET  ADO.NET  Base Class Library (BCL)  Remoting  Silverlight  TPL 



WCF  WCS  WPF  WF



Software factories



EFx Factory  Enterprise Library  Composite UI  CCF  CSF



IPC



MSRPC  Dynamic Data Exchange (DDE)   Remoting  WCF



Accessibility



Active Accessibility  UI Automation



Text and multilingual



support



DirectWrite  Text Services Framework  Text Object Model  Input method editor  Language Interface Pack  Multilingual User Interface  Uniscribe



v  d  e



Microsoft Windows family



Versions  Comparison  Components  History  Timeline  Criticism



Original



For DOS



Windows 1.0  Windows 2.0  Windows 2.1x  Windows 3.0  Windows 3.1x



Windows 9x



Windows 95  Windows 98 (development)  Windows Me



Windows NT



Early versions



Windows NT 3.1  Windows NT 3.5  Windows NT 3.51  Windows NT 4.0  Windows 2000



Client



Windows XP (editions  development)  Windows Vista (editions  development)  Windows 7 (editions  development)



Windows Server



Server 2003  Server 2008 (2008 R2)  HPC Server 2008  Home Server  Small Business Server  Essential Business Server



Specialized



Windows Embedded  Windows PE  Windows Fundamentals for Legacy PCs



Windows CE



Windows CE 3.0  Windows CE 5.0  Windows CE 6.0  Windows Phone



Upcoming



Windows Phone 7 Series  Windows 8  Windows Home Server V2



Cancelled



Cairo  Nashville  Neptune  Odyssey



Related topics



Windows Setup  OS/2



v  d  e



Microsoft



Board of directors



Steve Ballmer  James Cash, Jr.  Dina Dublon  Bill Gates  Raymond Gilmartin  Reed Hastings  Maria Klawe  David Marquardt  Charles Noski  Helmut Panke  Jon Shirley



Desktop software



Windows (components)  Internet Explorer  Office  Visual Studio  Security Essentials  Expression  Dynamics  Money  Encarta  Student  Math  Works  MapPoint  Virtual PC  Forefront  Home  Flight Simulator  Bob



Server software



Windows Server  SQL Server  IIS  PWS  Exchange  BizTalk  Commerce  ISA Server  System Center  Home Server  SharePoint (WSS, MOSS, Search Server)  OCS  Terminal Services  Microsoft Host Integration Server



Technologies



Active Directory  DirectX  .NET  Windows Media  PlaysForSure  App-V  Hyper-V  Silverlight  Windows Phone  Windows Embedded  Mediaroom  HDi



Web properties



Websites



adCenter  Bing  Channel 9  CodePlex  HealthVault  Ignition  Microsoft Store  MSDN  MSN (Games  msnbc.com  ninemsn)  TechNet  Windows Live (Groups  Hotmail  ID  Messenger  Spaces)



Live



Games for Windows Live  Xbox Live (Arcade  Marketplace)  Zune Social



Gaming



Microsoft Game Studios  Zone  XNA  Xbox  Xbox 360  Games for Windows



Hardware



Surface  Zune (4 / 8 / 16  30  80 / 120  HD)  MSN TV  Natural Keyboard  Jazz  Keyboard  Mouse  LifeCam  LifeChat  SideWinder  Ultra-Mobile PC  Fingerprint  Audio System  Cordless Phone  Pocket PC  RoundTable  Response Point  Venus (cancelled prototype)



Education and



recognition



MCPs  MSDNAA  MSCA  Microsoft Press  Microsoft MVP  Student Partners  Research  Studies related to Microsoft



Licensing



Client Access License  Shared source  Licensing Services



Criticism



Windows  Windows Vista  Windows XP  Windows 2000 (section)  Windows Me (section)  Windows 9x (section)  Office (section)  Xbox 360  Internet Explorer (section)  Refund



Litigation



Alcatel-Lucent v. Microsoft  European Union Microsoft competition case  United States v. Microsoft  Microsoft v. Lindows  Apple v. Microsoft  Microsoft vs. MikeRoweSoft



Acquisitions



Altamira Software  aQuantive  Azyxxi  Blue Ribbon Soundworks  Bungie  Calista Technologies  Colloquis  Connectix  Consumers Software  Danger  Farecast  FASA Studio  Fast Search & Transfer  Firefly  Forethought  GIANT Company Software  Groove Networks  Hotmail  Jellyfish.com  LinkExchange  Lionhead Studios  Massive Incorporated  Onfolio  PlaceWare  Powerset  ProClarity  Rare  ScreenTonic  Teleo  Tellme Networks  Vermeer Technologies  Visio Corporation  VXtreme  WebTV Networks  Winternals  Yupi



Annual Revenue: $60.420 billion USD (2008)  Employees: 89,809 (2008)  Stock Symbol: MSFT  Website: microsoft.com



Further information: List of assets owned by Microsoft Corporation



v  d  e



Windows Live



Web services



Account  Admin Center  Alerts  Calendar  FrameIt  Gallery  Groups  Home  Hotmail  ID  People  Photos  Profile  SkyDrive (Office Web Apps)  Spaces  Video Messages  Mobile



Software applications



Essentials  Family Safety  Mail  Messenger (Call)  Movie Maker  Photo Gallery  Sync  Toolbar  Writer



Developer services



Live Services



Microsoft Live Labs



Photosynth  Pivot  Seadragon (DeepZoomPix)



Discontinued services



Windows Live



Agents  Barcode  Events  Expo  Favorites  OneCare (Safety Scanner)  Personalized Experience  QnA  Search Center  Shopping  TV  Web Messenger  WiFi Center & Hotspot Locator



Microsoft Live Labs



Deepfish  Listas  Volta



v  d  e



.NET Framework



Architecture



Base Class Library  Common Language Runtime  Code Access Security  Assembly  Metadata  COM Interop



Common Language



Infrastructure (CLI)



Common Language Infrastructure  Common Type System  Common Intermediate Language  Virtual Execution System



CLI Languages



Common1



C#  Visual Basic .NET  C++/CLI (Managed)  F#   J#   JScript .NET  Windows PowerShell



Other2



Axum  A#  Boo  M  Oxygene   IronScheme (IronLisp)  IronPython  IronRuby  Nemerle  Phalanger  P#



Components



ADO.NET (Entity Framework  Data Services)  ASP.NET (AJAX  MVC  Dynamic Data)  Language Integrated Query  CardSpace  ClickOnce  Communication Foundation  Dynamic Language Runtime  Forms  Presentation Foundation  Remoting  Workflow Foundation   XAML



Other implementations



Mono  Compact Framework  XNA Framework  Micro Framework  Portable.NET  Silverlight  SSCLI  DotGNU



Comparisons



Java and C#  C# and Visual Basic .NET  Java and .NET platforms



Upcoming



"Acropolis"  "Jasper"  Parallel Extensions  Managed Extensibility Framework



1 Languages that are, will be, or have been included with Visual Studio or Microsoft Windows.



2 Non-academic or research languages with relatively large user-bases.



v  d  e



Microsoft development tools



Development



environments



Visual Studio (Team System  Express)  Visual Studio Tools for Office  Visual Studio Tools for Applications  Expression Web  Expression Blend   Microsoft Office SharePoint Designer



Languages and



compilers



Visual Basic (VB.NET)  Visual C++  Visual C#  Visual J#  Visual FoxPro  IronPython  IronRuby  PowerShell  XAML  F#



Major APIs and



frameworks



Windows API / SDK  .NET Framework  Managed code frameworks (Presentation  Communication  Workflow  CardSpace)  Silverlight  XNA  ASP.NET (ASP.NET AJAX)  LINQ  ADO.NET  DirectX  Robotics Developer Studio



Database tools



and languages



SQL Server



Express  Compact  Management Studio  Business Intelligence Development Studio  MSDE



SQL services



Analysis  Reporting  Integration  Notification



Other



Visual FoxPro  Microsoft Access  Extensible Storage Engine



Source control



Team Foundation Server  Visual SourceSafe



Driver development



Windows Driver Model  Windows Driver Foundation (KMDF  UMDF)  Windows Driver Kit  Windows Logo Kit



v  d  e



Microsoft Office



Office suites



Windows



95  97  2000  XP  2003  2007  2010 (upcoming)



Mac OS



98  2001  X  2004  2008  2011 (upcoming)



Applications



Desktop



Access  Communicator  Entourage  Excel  Excel Viewer  Groove  InfoPath  InterConnect  OneNote  Outlook (Outlook Connector  Add-ins)  PowerPoint  PowerPoint Viewer  Project  Publisher  SharePoint Designer  Snapshot Viewer for Access  Visio  Word  Word Viewer



Server



Communications Server  Forms Server  Groove Server  PerformancePoint Server  Project Server (Project Portfolio Server)  Office SharePoint Server (Excel Services  InfoPath Forms Services)



Online



Live Meeting  Office Live  Office Online  Office Web Apps  Outlook Web Access



Mobile



Office Mobile



Office tools



Clip Organizer  Graph  Equation Editor  Picture Manager  WordArt



Discontinued



Accounting  Binder  Data Analyzer  Document Imaging  Document Scanning  FrontPage  Liquid Motion  Mail  Office Assistants  PhotoDraw  Photo Editor  Schedule+  Vizact  Web Components



Related



Microsoft Works  MapPoint  Microsoft SharePoint  Windows SharePoint Server



Technologies



Information Bridge Framework  Object Linking and Embedding  Office Open XML  Office XML formats  Smart tags  Visual Basic for Applications  Clippy



History of Microsoft Office



v  d  e



Microsoft Expression Studio



Web  Blend  Design  Media  Encoder



v  d  e



Microsoft Windows components



Core



Aero  AutoPlay  AutoRun  ClearType  Desktop Window Manager  DirectX  Explorer  Taskbar  Start menu  Shell (Shell extensions  namespace  Special Folders  File associations)  Search (Saved search  IFilter)  Graphics Device Interface  Imaging Format  .NET Framework  Server Message Block   XML Paper Specification  Active Scripting (WSH  VBScript  JScript)  COM (OLE  OLE Automation  DCOM  ActiveX  ActiveX Document  COM Structured storage  Transaction Server)  Previous Versions  Win32 console



Management



tools



Backup and Restore Center  command.com  cmd.exe  Control Panel (Applets)  Device Manager  Disk Cleanup  Disk Defragmenter  Driver Verifier  Event Viewer  Management Console  Netsh  Problem Reports and Solutions  Sysprep  System Policy Editor  System Configuration  Task Manager  System File Checker  System Restore  WMI  Windows Installer  PowerShell  Windows Update  WAIK  WinSAT  Windows Easy Transfer



Applications



Calculator  Calendar  Character Map  Contacts  DVD Maker  Fax and Scan  Internet Explorer  Journal  Mail  Magnifier  Media Center  Media Player  Meeting Space  Mobile Device Center  Mobility Center  Movie Maker  Narrator  Notepad  Paint  Photo Gallery  Private Character Editor  Remote Assistance  Windows Desktop Gadgets  Snipping Tool  Sound Recorder  Speech Recognition  WordPad



Games



Chess Titans  FreeCell  Hearts  Hold 'Em  InkBall  Mahjong Titans  Minesweeper  Pinball  Purble Place  Solitaire  Spider Solitaire  Tinker



Kernel



Ntoskrnl.exe  hal.dll  System Idle Process  Svchost.exe  Registry  Windows service  Service Control Manager  DLL  EXE  NTLDR / Boot Manager  Winlogon  Recovery Console  I/O  WinRE  WinPE  Kernel Patch Protection



Services



BITS  Task Scheduler  Wireless Zero Configuration  Shadow Copy  Error Reporting  Multimedia Class Scheduler  CLFS



File systems



NTFS (Hard link  Junction point  Mount Point  Reparse point  Symbolic link  TxF  EFS)  FAT32FAT16FAT12  exFAT  CDFS  UDF  DFS  IFS



Server



Domains  Active Directory  DNS  Group Policy  Roaming user profiles  Folder redirection  Distributed Transaction Coordinator  MSMQ  Windows Media Services  Rights Management Services  IIS  Terminal Services  WSUS  Windows SharePoint Services  Network Access Protection  PWS  DFS Replication  Remote Differential Compression  Print Services for UNIX  Remote Installation Services  Windows Deployment Services  System Resource Manager  Hyper-V



Architecture



NT series architecture  Object Manager  Startup process (Vista/7)  I/O request packet  Kernel Transaction Manager  Logical Disk Manager  Security Accounts Manager  Windows File Protection / Windows Resource Protection  LSASS  CSRSS  SMSS  MinWin



Security



User Account Control  BitLocker  Defender  Data Execution Prevention  Security Essentials  Protected Media Path  Mandatory Integrity Control  User Interface Privilege Isolation  Windows Firewall  Security Center



Compatibility



Unix subsystem (Microsoft POSIX   Interix)  Virtual DOS machine  Windows on Windows  WoW64  Windows XP Mode



Categories: Microsoft Windows multimedia technology | Microsoft application programming interfaces | Multimedia frameworks | Windows Vista