Data-Centric Architecture: A Model for the Era of Big Data

Data-Centric Architecture: A Model for the Era of Big Data | Dr Dobb's Journal #container #header #nav { font-size: .9em; margin-top: 0px; margin-left: 5px; width: 999px; position: relative; top: 4px; } /* Absolutely positioned #nav is lower on homepage, see home.css */Subscribe
Newsletters
RSS
Search: Site Source CodeHomeArticlesNewsBlogsSource CodeDobbs on DVDDobbs TVWebinars & Events #nav .parallel a:hover { text-decoration: underline; }Go ParallelChannelsArchitecture & DesignC/C++DatabaseDevelopment ToolsEmbedded SystemsHigh Performance ComputingJavaJolt AwardsMobilityOpen SourceSecurityWeb DevelopmentWindows/.NETVisual Studio 2010
INFO-LINK

Architecture & Design ? Permalinkfunction emailLauncher(filePath,storyURL) {uri = filePath + encodeURIComponent(storyURL);window.open(uri,"","toolbar=no,scrollbars=auto,location=no,status=no,width=733,height=590,resizable=1");}Data-Centric Architecture: A Model for the Era of Big DataBy Rajive Joshi, Ph.D., March 26, 2011

0 Comments

As businesses struggle to handle large volumes of data rapidly, technologies used in industry and by the masters of the universe on Wall Street are suddenly becoming fashionable...again

Wired and wireless communication networks are making data collection and transmission cheap and widespread. In the future, networks will weave many devices and subsystems into complex integrated distributed systems that will become the fabric of business and daily life.

Building such distributed systems is far from simple, however. They must be assembled from independently developed software components. Integration, especially combined with real-time performance demands, becomes the key challenge.

This article outlines fundamental design principles that enable integrating distributed systems from components. I use a data-centric approach to this design, as the data is the key element that must flow through the various systems.

The key to data-centric design is to separate data from behavior. The data and data-transfer contracts then become the primary organizing constructs. With carefully controlled data relationships and timing, the system can then be built from independent components with loosely coupled behaviors. Data changes drive the interactions between components, not vice versa as in traditional or object-oriented design.

The resulting loosely coupled software components with data-centric interfaces are then integrated into a working system through a data bus. The data bus connects data producers to consumers and enforces the associated Quality of Service (QoS) contracts on the data transfers. This design technique is naturally supported by the Data Distribution Service (DDS) specification for real-time systems, which is a standard from the Object Management Group. Implementations of this standard are available from many vendors.

The techniques described here are proven in hundreds of mission-critical applications including robotics, unmanned vehicles, medical devices, transportation, combat systems, finance and simulation. A Future Distributed System

To understand the dynamic nature of next-generation distributed systems, it is helpful to examine a representative scenario: an air traffic control system. The future air traffic control integrates a variety of disparate systems into a seamless whole — a system of systems. On the edge is a real-time avionics system inside the aircraft. The control tower in the center of the figure communicates with the avionics system, and then out to the enterprise data servers at the airport. The system thus comprises connectivity from the "edge" (devices) to the "enterprise" (infrastructure services).

The data in the avionics system flows at high rates and is time-critical. Violating timing constraints could result in the failure of the aircraft or jeopardize safety. Although aircraft traditionally operate as independent units, future aircraft must integrate closely with automated traffic control and ground systems.

The control tower is another independent real-time system; it monitors various aircraft in the region, coordinates their traffic flow and generates alarms to highlight unusual conditions. The data flowing in this system is time-sensitive for proper local and wide-area system operation. However, the system may have a greater tolerance for delays than the avionics systems.

The control tower communicates with the airport's enterprise information systems, which track flight status and other data and may communicate with multiple control towers and other enterprise information systems. It is also responsible for synthesizing a dashboard view containing passenger, flight arrival and departure status information. Because it is not in the time-critical path, the enterprise information system can be more tolerant of delays than other systems.Key Design Challenges

This so-called "system-of-systems" must deal with a many issues, such as correctly handling myriad differences in data exchange, performance, and real-time requirements. The architecture also involves different technology stacks, design models, and component lifecycles.

To support system growth and evolution, the integration must be robust enough to handle changes on either side of an interface. To do this, only minimal assumptions should be made about the interfaces between systems—the interface specifications should describe only the invariants in the interaction. Behavior can then be implemented independently by each system; the interface between them should not include any component-specific state or behavior. This avoids tight coupling.

The systems on either side of an interface may differ in quantitative aspects of their behavior, including differences in variations in data volumes, rates, and real-time constraints and so on. The term "impedance mismatch" is shorthand for all the non-functional differences in the information exchange between two systems. Critically, a developer can capture these non-functional aspects of the information exchange by attaching QoS attributes to the data transfer. When QoS terms are explicit, responses to impedance mismatches can be automated, monitored, and governed.

1 2 Next function setThreadView(view){var threadview = document.getElementById('treeviewcomments');var flatview = document.getElementById('listviewcomments');if(view=='thread'){ threadview.style.display='inline'; flatview.style.display='none';}else{threadview.style.display='none';flatview.style.display='inline';}}img.comment-tree {border:medium none !important; float:left !important; height:45px !important; margin:0 2px 0 0 !important; padding:0 !important; width:20px !important;}.top-comment {background-color:#EEEEEE;clear:both;padding:10px 10px 10px 23px;font-size:0.9em;}#content .top-comment h1 {color:#999999;font-size:1.2em;margin:0 0 10px;padding:0;}#comments .top-comment h2 {clear:both;font-size:1.7em;}.top-comment {background-color:#EEEEEE;clear:both;padding:10px 10px 10px 23px;font-size:0.9em;}#content .top-comment h1 {color:#999999;font-size:1.2em;margin:0 0 10px;padding:0;}#comments .top-comment h2 {clear:both;font-size:1.7em;}



Care to Comment?

Subject (max length: 75):

Comments:

Captcha:

Type the characters you see in the picture above.

Around the Web Erlang vs. Java Memory Architecture

Erlang's actor model gives it a very different memory architecture from Java.

Quick Read

Building a Queue With Two Stacks

How to use pair of stacks to build a queue in the same way they're created in functional languages.

Quick Read

Mercurial vs. Git

One developer's experience using the two primary DVCSs.

Quick Read

Writing Games for Windows Phone 7

A detailed how-to article with code and illustrations.

Quick Read

How Hadoop MapReduce Works

An inside view of Apache Hadoop's built-in command center and how it manages resources.

Quick Read

Xcode 4 Ships! Is It Worth Upgrading?

The latest release of Apple's IDE analyzed in great detail.

Quick Read

Architecture & Design Recent ArticlesPragmatic DevelopmentFast, Parallelized CRC Computation Using the Nehalem CRC32 InstructionThis Week's Developer Reading ListQt Application Development for SymbianAgile at 10: What We Believe (Scott Ambler)Most PopularStoriesBlogsGetting Started with The Cloud: The EcosystemPragmatic DevelopmentIntroduction to OpenMPThe Boost.Threads LibraryMaking Pimpl Easy New Intel Turbo Boost Technology Evolving Architectures: Part VII Software Development Lifecycle, Fahgettaboudit! Implementing Thread Local Storage on OS X Think Parallel, Think Java Video View All VideosMost Recent Premium ContentWhitepapersTaneja Group: The Requirements for Enterprise Storage in the CloudsFive Steps to Managing Reference Data More EffectivelyDoes Your Risk Management Approach Pass the Sink or Swim Test?This month's Dr. Dobb's Journal

This month, in this special April issue of Dr. Dobb's Journal, we examine the state of the agile movement, explore the Qt development framework, look back at AntiPatterns, and much more!

Download the latest issue today. >>

Enabling People and Organizations to Harness the Transformative Power of TechnologyCIOs & IT ProfessionalsBlack HatBYTECloud ConnectDark ReadingEnterprise 2.0Enterprise ConnectEnterprise EfficiencyHDIInformationWeekInformationWeek 500InformationWeek 500 ConferenceInformationWeek AnalyticsInformationWeek EventsInformationWeek Global CIOInformationWeek HealthcareInformationWeek IndiaInformationWeek SMBInteropNetwork ComputingNo JitterPlug into the CloudTechWeb.comThe BrainYardSoftware DevelopersDr. DobbsDr. Dobbs M-DevDr. Dobbs DigestDr. Dobb's UpdateTechWeb.comWeb & Digital ProfessionalsInternet EvolutionWeb 2.0 ExpoWeb 2.0 SummitTechWeb.comGovernment OfficialsGTEC DCGTEC OttawaInformationWeek GovernmentTechWeb.comVertical MarketsAdvanced TradingBank Systems & TechnologyCreateYourNextCustomerInformationWeek GovernmentInformationWeek HealthcareInsurance & TechnologyLight Reading / TelecomThe CMO SiteWall Street & TechnologyGame Industry ProfessionalsGamasutra.comGame Developers Conference (GDC)Independent Games FestivalGame Developer MagazineGDC EuropeGDC ChinaGame Career GuideGame Advertising OnlineGlobal Communications
Service ProvidersHeavy ReadingHeavy Reading InsidersPyramid ResearchLight ReadingLight Reading MobileLight Reading CableLight Reading EuropeLight Reading AsiaEthernet ExpoTelcoTVTower SummitLight Reading Live & Virtual EventsWebinarsMost PopularBob Evan's Global CIOCable CatchupCloud Connect BlogDigital LifeEvil BytesInformationWeek AnalyticsInterop BlogMonkey BidnessOver the AirPersonal TechThe PhilterValley WonkUBM TechWeb Reader ServicesAbout UBM TechWebAdvertising ContactsTechnology Marketing SolutionsContact UsFeedbackReprintsTechWeb Digital Library / White PapersTechWeb Events CalendarTechWeb.com Terms of Service | Privacy Statement | Copyright © 2011 UBM TechWeb, All rights reserved.