Automotive onboard navigation systems have become an integral part of today’s vehicles. Advanced navigation systems combine GPS data, digital satellite imagery, real-time traffic updates, and digital map data with points of interest information to produce dynamic three dimensional map views and a superior driving experience. These enable drivers to better visualize where they are in order to make driving and navigating easier. Maps and traffic information are displayed on a navigation screen in the center of the dashboard. The system can also display current vehicle location, traffic flow, incident reports and warnings along the current driving route.

Drivers who need guidance can ask the navigation system for directions by supplying street names or points of interest. An on-board text-to-speech engine takes the resulting route information and converts it into a voice to give the driver turn-by-turn directions.

The software that powers these navigation systems represents a new class of automotive applications that is best characterized as data-driven. Regardless of where you find them, data-driven software applications tend to have the following common characteristics:

• They process and integrate information derived from diverse sources including sensors, stored media (disks, CDs, DVDs), satellite feeds, cellular connections and other onboard applications located elsewhere.
• Data tends to move through the system in the form of dynamic streams.
• Monitoring and management of data streams can extend over a period of time as opposed to only the most recently needed records.
• There is a need to take autonomous action based on the state of the data being monitored, such as changing a display, signaling a malfunction or updating turn-by-turn directions for the driver.
• The data is subject to a large number of continuous queries and computationally complex operations.
• The application has time-sensitive or real-time requirements.

Consider the earlier example of a navigation system. The resulting electronics components consist of a processing system, a GPS unit, a DVD or HDD with digital map info and points of interest information, and a receiver system which collects real-time traffic information and music from a satellite or digital radio connection.

During operation, the receiver must decode both music and traffic-related information streams. It then passes the music to the car's radio subsystem and relays the traffic to the navigation system in the form of a stream of records. The navigation system must filter out decoded information that is not appropriate for the vehicle’s current location - which is determined through communication with the GPS unit. Once the navigation system determines its location, it identifies the locally relevant traffic information and sends an output stream to the display for presentation to the driver. Simultaneously, the system is also busy calculating the shortest route to the destination. Whenever the navigation system is on, it is constantly sampling a variety of data streams from multiple, distributed sources, looking for information that is relevant to its location and the driver’s requests, and integrating and processing that data to produce an action.

Modern vehicles produce very large amounts of data from a variety of sources. This data needs to be acquired, filtered, processed, integrated and managed in order for data-driven applications to analyze operating conditions and react to their environment. Clearly, the ability to apply traditional data-processing languages and operators is highly desirable because of their inherent efficiency in solving data-related problems. Unfortunately, the traditional database assumption about the static nature of data sources does not apply to the dynamic streams nature of most automotive systems. The question becomes: how best to design an automotive data management system to enable multiple subsystems and applications to share information and increase functionality to provide more benefits to drivers – all without increasing development costs, manufacturing costs or risk?

The majority of commercial data management products have evolved from enterprise level data processing systems. They are organized around a passive database model that is designed to manage a large collection of persistent data either on a central disk or in main memory. Traditional database-oriented applications are built around a “store-then-process” model. When data arrives from a remote source it must first be stored in database before it can be accessed and processed. 

There are two primary differences between automotive-based data sources and traditional database sources. First, vehicles typically deliver data in the form of Streams. That is, they produce data autonomously and continuously at regular, well-defined time intervals. The processing of these streams needs to occur rapidly, in or near real-time. This is partly because it is extremely expensive to save raw data to permanent storage and partly because the data streams represent real-world events like changing GPS coordinates or out-of-tolerance operating conditions that must be responded to immediately.

The second major challenge is that the electronic subsystems found in most vehicles are fundamentally different from the resource-rich computing environments typical of an enterprise-oriented database system. Total system memory in a vehicle navigation system is severely limited compared to memory requirements of a typical database. Navigation systems seldom exceed tens of Mbytes of system memory. The embedded microprocessors used in automotive systems tend to integrate more system-on-a-chip functionality but typically provide far less computing power than their enterprise server equivalents. Plus, database technologies typically require large amounts of persistent storage, far exceeding the storage capacities of FLASH or NVRAM. Until recently, hard drives have been poorly suited for automotive environments due to the extreme range of operating temperatures and vibration. And while improvements in hard drive technology are making them more a more feasible option, they remain best for storing persistent data such as map data and do not serve as a repository for live stream data.

Clearly, a different approach is required. One that preserves the ability to apply traditional data-processing languages, relational data models and operators to stream-oriented data sets and at the same time one that overcomes the performance and cost issues associated with classic database systems.

Today, data management and integration code typically makes up 50% or more of an application. Programming data functions manually in C or C++ has become very complex, time-consuming and error-prone. Moreover, hand-coded data management routines in C/C++ often suffer from implementation inconsistencies that make applications harder to debug, integrate and evolve.

According to studies performed by Software Productivity Research (SPR), data processing languages, like SQL, are ten times more expressive than C or C++ for data-oriented tasks. Even the best C programmers typically need more than 100 source statements to code a complex data function. By comparison, a SQL programmer needs just six. Greater language expressiveness reduces coding errors and improves developer productivity. This reduces development costs, improves quality, and shortens time-to-market. The higher level of abstraction also provides improved code portability, greater code reuse and reduced cost of maintenance. 

Encirq has pioneered an exciting new category of device data management technology designed specifically for automotive applications that have the need to acquire, integrate, manage and act upon diverse and dynamic data sets. The Encirq approach assumes that the relationships between the data, the processing algorithms and the queries to be executed are quite complex and subject to continual evolution. Furthermore, it assumes that the processing requirements will vary as the application evolves over time and that many of the data sources will be distributed and under the control of diverse, heterogeneous subsystems. Most importantly, the assumed point of data integration is within the application where it is needed – not within a central database.

The Encirq DeviceSQL Framework consists of two main elements: a development environment and a performance-optimized platform environment. 

The development environment is used to create a description of the application and its semantics. It incorporates a SQL-baed language that is optimized for data operations and includes extensions specifically for the unique needs of devices. It also includes tools for automating the generation of data management code from DeviceSQL statements into portable ANSI C format for compilation and direct, inline execution with the rest of the application.

The target platform execution environment provides various data management services and helps implement a high performance data stream processing engine that applications may draw on as they execute. The execution environment is typically implemented in less than 48K bytes of memory.

The DeviceSQL Framework provides unmatched flexibility, control and performance for automotive application designers.

1. It supports operations on dynamic stream data as well as table data
2. There are no proprietary APIs or forced architectural implementations
3. It works seamlessly with existing C/C++ and Java applications and development environments
4. It supports virtually any microprocessor architecture and target operating system for maximum portability and reuse

Maximizing Computational Capabilities

Automotive applications tend to be computationally intensive. The ease and completeness with which complex information can be modeled is described in two distinct dimensions referred to as “structural” and “semantic” expressibility.

The structural expressibility of the system is determined by the richness of both the primitive data types and the constructors that can be used to form new complex types from the primitives. In traditional database systems the primitives are usually defined by the SQL standard. For example, strings, numbers, dates, etc. The type constructors are typically “flat” records and tables. All system information must be mapped into these flat constructs.

The semantic expressibility of a system reflects the ability to describe the meaning and relationship among information. In general, semantic descriptions can be used to constrain the behavior of the system For example, one can choose to constrain a system to prevent the adding of inches to feet or one can model the system to facilitate the addition of inches to feet without the need for complex conversion at the application level. Standard database systems provide very little support for semantic expressibility. 

By contrast, DeviceSQL delivers the highest level of computational capability by maximizing both the structural and semantic expressibility of a system. The approach goes beyond the flat model imposed by most database systems and includes type constructors for multilevel and recursive record types, as well as for set components. Of the types supported in the DeviceSQL environment, Streams represent an exciting new category of capabilities that are ideally suited for automotive systems.

As noted earlier, automotive electronics typically produce and deliver information in the form of data streams. Streams can exist in a number of different forms. For example, data streams can be used to manage time-sequenced information from sensors, transducers and network connections. Record Streams can be used to manage structured information such as the result of query operations on stored data sets or accessing files or tables. Streams provide a very efficient way of operating in the distributed, resource-constrained environments that are typical of automotive systems.

With DeviceSQL, every primitive data type has a corresponding Stream type. Streams can be created, detected, merged, aggregated, routed and destroyed. A Stream operator is a function which accepts one or more Streams as an input and produces a corresponding Stream as output. Steam operations include filtering, appending, merging, finding intersections, and aggregating. Here are some highly simplified examples of stream operators. For the following examples assume that S is a Stream of records which contains the integer values {1,2,3,4,5,6,7,8,9,10}:

• The operation mapStream(S,square) will produce the resulting Stream {1,4,9,16,25,36,49,64,81,100}.
• If p is the operation EVEN then filterStream(S,EVEN) will produce the resulting Stream: {2,4,6,8,10}
• If S1 is the Stream {1,3,5,7,9} and S2 is the Stream {2,4,6,8,10} then the operation appendStream(S1,S2) will produce the sequence {1,3,5,7,9,2,4,6,8,10}.
• If S1 is the Stream {1,3,6,8} and s2 is the Stream {2,4,5,9} then `mergeStream(S1,S2, Identity) will produce the Stream {1,2,3,4,5,6,8,9}.
• If s1 is a Stream containing {1,3,4,5,7,8} and s2 is a Stream containing {1,2,3,6,8,9} then intersectStreams(s1,s2,Identity) will produce the Stream {1,3,8}. 

A Stream-based system processes incoming data streams as defined by the application. This is fundamentally a data-flow system. Data literally flows through a sequence of processing operations from input to output. Input Streams can be queried, filtered, merged and operated on in the same fashion as any other data type as specified by the application. Ultimately, output streams are presented to the application or forwarded as an input stream to another system, subsystem or application. The basic components of a Steam-based system are illustrated below.

The application defines the operations that need to be performed on the application data. The application interfaces with a run-time Stream Processing Engine that is responsible for providing the underlying services for both stream-based data and persistent data. The services include various storage managers and adaptors for a number of different media including random access memory, disk drives, file system, FLASH memory, NVRAM and even sensors and network connections. The storage management services provide all of the traditional database capabilities including Transactions, Recovery, Rollback, and Commit logic. Memory management and indexing services are also provided. Only those services needed by the application are included in the runtime.  Although the internal architecture is significantly different from a traditional database system, the relational data interface is completely preserved. This gives the ability to apply traditional data-processing constructs and operators, which is highly desirable because of their logical completeness and familiarity.

The Stream Engine overcomes many of the performance issues associated with classic database systems:

1. The server must include the SQL query parser, query plan generator and interpreter as part of the run-time system.
2. The server is forced to include the code necessary to support all possible type and query operations as part of the runtime system – even though only a small subset may actually be executed during the life of the system. This increases code size, reduces system performance and adds expense.
3. Most expressions (even purely arithmetic ones) must be evaluated by out-of-line function calls. Since most data scans involve simple arithmetic calculation, the cost of interpreting versus compiling significantly reduces system performance.

With DeviceSQL, the entire data management portion of the application is analyzed and compiled into native machine code. The Stream Engine does not require an SQL parser, query generator or interpreter of any kind (though a DeviceSQL interpretive API is provided as an option) . SQL operations are analyzed for syntactic and semantic errors at build time. This enables the entire runtime environment to execute in less than 48Kbytes of memory. The result is a significant improvement in system performance and responsiveness with optimized system resource utilization to reduce costs.

The DeviceSQL programming model is nearly identical to the programming model for database operations. All of the standard SQL constructs are preserved which enables developers to build more sophisticated applications with far less coding effort and in less time.  

Consider the following example. A navigation application must locate a McDonald’s restaurant within 3 kilometers of the vehicles current location. This example uses two types of streams. The first is a data stream representing current location, specified by the GPS sensor (stLocation). The second is a record stream of Point of Interest (stPOI) information derived from a CD stored in the audio entertainment system. The operation is expressed straightforwardly as a traditional SQL SELECT statement:

Note that in this example there are no data tables – only Streams. Another important attribute of Streams is that they compute only as needed. The internal representation of a Stream only requires space enough for the methods; including any shared local variables used to define the Stream. For example, an appendStream operation does not need to copy the elements of Stream1 and Stream2 to a temporary buffer before returning a result. This is critically important to automotive applications in that it enables extremely large data streams to be represented and operated on within very small amounts of local storage and with ultra fast performance. Additionally, the highly simplified programming effort leads to faster time to market, higher quality software and much greater application portability.

Use of Streams in Automotive Systems

Streams form the basis for inherently scalable and portable automotive applications. The possible uses for Stream-based data management systems are almost limitless. They include:

• Integrating data packets to form and deliver messages on the internal vehicle network
• Gathering and aggregating personalization information from vehicle subsystems
• Integrating point-of-interest information with real-time traffic data for the navigation system
• Monitoring and analyzing operational conditions
• Message queue management
• Predictive failure analysis and vehicle diagnostics

Automotive applications are being transformed by the need to bring increasingly larger and larger volumes of digital data into the modern vehicle. Data is produced continuously and autonomously in the form of streams which need to be processed in or near real-time. The amount of software required to perform the data management and integration tasks already constitutes 50% or more of an application and it is increasingly difficult to optimize data management code with prevailing techniques and technologies.

Until today, automotive software developers have had to choose between two approaches when building applications. They could choose to hand code the data functions manually in C or C++.  This approach is complex and time-consuming. It also leads to applications that are harder to debug, integrate and evolve. Alternatively, they could choose to implement an embedded database.  The drawback here is that database applications are built around a “store-then-process” model that is not well suited for automotive applications that have to process streaming data in near real time. Additionally, the system resource requirements needed to implement a database impact run-time performance and manufacturing cost.

With DeviceSQL, Encirq has pioneered an exciting new category of device data management technology designed specifically for automotive applications that provides the best of both worlds. The Encirq DeviceSQL Framework integrates the ability to apply traditional data-processing languages and operators to automotive data sets. The higher level of data abstraction reduces application complexity and improves productivity. This leads to reduced development costs and faster time-to-market.

The DeviceSQL approach to processing stream data overcomes the performance and resource issues associated with traditional database systems. Streams provide a very powerful and efficient way of operating in the resource-constrained, distributed data environments that are typical of automotive systems. They form the basis for a new generation of scalable, portable and innovative automotive applications.

Innovation is the lifeblood of today’s leading automotive manufacturers. With DeviceSQL, organizations can focus less on the underlying low-level details of implementing complex data management routines. Instead, DeviceSQL enables them to focus on delivering innovative new data-driven features and services which unlock the value of the data that is available to them – and in doing so to create new consumer experiences, new business models and new revenue streams.

Encirq Corporation is the global leader in device data management software and the creator of DeviceSQL^™, the software framework for device data management. Encirq enables device manufacturers and after-market application providers to employ device data in more efficient and innovative ways to increase their products’ independent intelligence, performance and usability. Encirq customers gain a competitive edge in their capability to deliver superior products, improved services, innovative business models and more compelling user experiences all while improving device performance, optimizing system resources and reducing costs.

Encirq is headquartered in Silicon Valley with offices in Japan and Korea.