In a new article for insideBIGDATA, Moshe Kranc, chief technology officer of Ness Digital Engineering, discusses the major benefits and limitations that can help organizations decide if a Data Vault would meet their specific data architecture needs given that data vault modeling requires considerable experience if used. “Data Vault architecture is an innovative, hybrid approach that combines the best of 3rd Normal Form (3NF) and dimension modeling,” notes Moshe.
Month: July 2017
Ride the Next Wave of Growth with Personalized Learning Experiences- Education Sector Series: Part 2
The education industry is poised for a technology revolution. A 2016 report on EdTech Spending states that the global education technology market is expected to grow at 17% to over $250B by 2020. This marks a huge opportunity for education technology services companies to accelerate growth by offering innovative learning solutions and platforms that enable personalized learning experiences.
However, a major challenge for these companies is to be able develop new products, and maintain the right balance between modernizing existing revenue-generating products and developing new ones for business expansion– and getting them to market quickly.
Ness works with some of the largest education technology companies and helps them create a roadmap for accelerating their pace of change with platforms focused on delivering improved learning experiences and successful outcomes. Here are some of the examples of our recent work in the education sector:
Measuring Learning Efficacy: To maximize learning and identify improvement areas, Ness helped a client develop a suite of tests that assess knowledge in math, reading and writing for students preparing to enroll in college-level courses. The product is used to identify students’ strengths and weaknesses in each subject area and to help them improve their skills by presenting relevant, interactive content through online learning tools.
New Learning Product for Professionals: Ness helped a client significantly improve user satisfaction with a new interactive, learning solution for technology professionals. Ness developed a simulation that enables network administrators to practice creating network solutions via hundreds of structured lab scenarios, including detailed instructions, topology diagrams, critical-thinking questions, hints, and answers.
Improving Product Accessibility and Scale: For a leading virtual school and learning platform provider with millions of active users, Ness developed a mobile app for its flagship product and transformed its customer-facing Web platform. Ness helped another client improve the overall user experience, availability and response time of an online learning platform which resulted in increased customer satisfaction and enabled the product to handle 200% more concurrent users. Ness also has extensive experience in software quality and accessibility testing.
The future of education looks radically different in the next five years. Technology advancements like augmented reality, machine learning etc. backed by consumer preference for on-demand information access and personalized experiences, and a new workplace culture that demands continuous learning makes it extremely vital for education technology providers to look for innovative product offerings to help their customers connect with a different generation of learners.
Working with a partner like Ness which has the domain expertise and digital technology capabilities can help you stay ahead of your competitors in the transforming education landscape and stay closely aligned with the changing needs of the industry.
Read more about how Ness can help you create a roadmap to drive the digital disruption in education. https://ness.com/work-in-action/industries/#1453153709885-0bd58f3e-83e5
Driving Innovation With Connected Cars
The automotive industry is transforming at an accelerated pace, and the Connected Car revolution is a major catalyst to this change.
With Over 380 million Connected Cars expected to be on the roads by 2021, automotive players are facing an urgent need to drive strategic technology innovations in areas, such as ADAS (advanced Driver Assistance Systems), in-car user experience and content services, data management and telematics.
Ness recently participated in the London Tech Connected Car Event, which brought over 600 automakers, tier 1 suppliers, insurers and technology providers under the same roof to discuss strategies needed to keep pace with an evolving automotive market.
Ness Executive Vice President, Pete Rogers, was interviewed at the London Tech Connected Car event. He provided insight into how connected cars are a highly complex digital platform, and how Ness’s strong expertise in this area can help automotive players prepare the roadmap to develop a Connected Car ecosystem, and build the technical prowess to lead the automotive digital disruption.
Are you scrambling for answers on what’s driving the Connected Car market, and how can you transform your business to leverage the opportunity? Read this excerpt from Pete’s interview.
Q. Pete, Ness has an interesting perspective on Connected Car ecosystem. How did your company become part of it?
Pete: It might not be popular with everyone in the industry, but we tend to think the Connected Car as software on wheels – and that makes it a true digital platform. Ness is a company that is expert in designing and building next-generation digital platforms across sectors as diverse as the Industrial Internet, Broadcast and finance, so it’s an area we are very excited about.
To be an expert you need to have the right skills in usability and a deep understanding of the inherent complexity and huge volumes of data involved. And as we are a partner to many already in the ecosystem, the Connected Car is an obvious trend for us to invest in and focus on.
Watch Video to listen to the full interview to uncover more insights on how automotive players can lead the Connected Cars revolution.
Key Steps in Eliminating the Roadblocks to DevOps Adoption
DevOps adoption is a strategic priority for most enterprises right now and the benefits that early adopters are reaping is testimony to its true potential in driving organizational change. But getting DevOps right can be a tough challenge for many organizations considering that it entails major transformations and requires a combination of people, processes and technologies to be in sync for the organization to be truly fast and agile.
Addressing the fundamental issues that can impact adoption is an important starting point. Additionally, it is important to be able to define the key metrics to track progress along the DevOps adoption journey.
In a new article from the Enterprisers Project, Amit Gupta shares his insights on the five key components to consider while embarking on the DevOps journey.
“The key is to have a few, simple metrics that truly reflect value to the end customer, including production deployment frequency; average lead time for a production change; average production recover time; and change failure rate in production. A true measure of successful DevOps adoption is an improvement in all these parameters over time,” notes Amit.
Read more https://enterprisersproject.com/article/2017/7/devops-lessons-learned-advice-it-leaders
Building a Data-driven and Artificial Intelligence Culture in the Enterprise: A Winning Formula
At Ness, our enterprise customers are eager to explore their hidden treasure in the data that they have generated over the years. Market research shows a growing demand for data monetization, and estimates that the Big Data market will be $125B (USA, including hardware, software and services) by 2025 [Source: Million Insights]. In 2014, Big Data generated $23B in revenue and we are expecting 5x-6X growth by 2025.
Here are additional data facts – According to Forbes, the cloud computing market is increasing with a CAGR of 19% to $162B by 2020. Forbes also reported that 1.5 MB of data is generated every second for every human being on the planet. This means the digital universe of data will stand at about 44 zettabytes in the next few years. That’s an incredible amount of data!
Digital leaders have already started harnessing data generated through systems of record and other semi-structured/unstructured data sources, carrying out exploratory analytics that assist them with business insights and perhaps making more informed decision when it impacts revenue growth and profitability. Examples are Adobe, Nike, Amazon, Google, Yahoo! etc.
One more data point comes from the retail industry which is undergoing severe disruption. Amazon has been sitting on a data treasure for a very long time. See how they have used to their advantage — today Amazon’s stock price is five times what it was in 2012. Others retail stores are filing for Chapter 11 and restructuring themselves. Why? – Because most of them did not read their customer’s pulse right. Most of the digital laggards among retailers are shutting down stores, reducing their footprint and going back to the drawing board to rework their digital strategy.
CEOs are looking at two simple indexes: How do I increase my revenue share and how do I increase my profitability? For this every CEO is now being asked by their shareholders to evaluate their digital strategy. Three key pillars to drive any CEO’s objectives are: Customer centricity, operational efficiencies and the ability for an organization to innovate by building new products more quickly and more efficiently. How do you achieve this? What are the key ingredients? Of course, DATA! Most enterprises have tons of data which can be aligned to the three pillars of digital transformation which I have outlined. The key is to build that as a culture in your organization.
At Ness, we recently invited some of the industry leaders to discuss how data and AI (Artificial Intelligence) can help them have an edge on their competitors and the key challenges/opportunities they perceive for their organization. We hosted a panel of Digital Leaders from Airbnb, Facebook and Lightspeed Ventures as well as representatives from several other leading organizations, to provide interesting insight on how data and AI are enabling disruption across industries.
Below I noted the questions we explored with a summary conclusion based on the group’s input:
- How are analytics used in any industry to create a competitive advantage?
Common Observation: Almost everyone in the panel and the audience agreed that creating a data driven culture is essential to creating a competitive advantage. Enterprises need to innovatively use their data to help them succeed.
- What are the barriers to entry for a company that wants to adopt best analytic practices?
Common Observation: Access to good data is very critical. This is where data preparation plays a key role. Applying AI on noisy data is not always helpful. Obviously, infrastructure is assumed. The group emphasized the importance of having a properly skilled team to implement data initiatives. Even though big data technology has been in existence for over a decade, enterprises still see a significant scarcity of skill availability to help them scale and overcome their data challenges. This includes selecting the right choices in hardware/software, understanding the use cases and most importantly, having noise free data.
- What tasks within your industry could be performed by AI? Do you expect to see AI-based products being deployed in the near term?
Common Observation: Everyone agreed that we will see an explosion in AI adoption for repetitive work which machine/computer systems learn and train to execute faster and with higher precision than humans. It will be used to reduce the cost and increase the quality of services. However, scientists and VCs feel AI implementation across the board is still several years away, and replacement of humans is still a myth. For example, analyzing videos to carry out predictive outcomes is still a non-trivial problem to be solved.
- Are you generally optimistic or pessimistic about the long-term future of Artificial Intelligence and its benefits for your industry? What impact will it have on employment in your organization?
Common Observation: It was agreed that it is difficult to predict benefits and negative impacts in simplistic terms. As the AI usage/implementation matures – there will be government regulations, policies, ethics and so many other factors which will start playing a role.
In conclusion, here are some important highlights and tips to get started with data monetization:
- Every enterprise today needs to have a defined data driven story. It needs to come from the top of the organization (CEO, Business Heads etc.). Having a CDO (Chief Data Officer or a Chief Digital Officer) is helpful to define a path to building a data driven organization.
- Bring your data into one single place. Do not shy away from bringing siloed organizations under one umbrella and start creating plans to have your data lake/reservoir in place. There are several facets to it and it needs to be executed correctly.
- Data Preparation: This in my opinion is the MOST important journey which every enterprise should undertake to harness good data for insights. Good data will differentiate you from others and will provide a significant competitive advantage.
- Data Skills: This is a very difficult challenge to solve. Encourage your engineers to explore, learn, and retrain themselves to understand available tools/technologies in the market. Consider working with partners that can provide the relevant skills. You can use them to build your own data engineering center of excellence
- Help your business team identify the right use cases that can be solved by technologies.
At Ness, we are here to help you build a data culture for your enterprise. We have experience doing this for several customers across verticals. We can help you too.
I love a quote from Sherlock Holmes (A Study in Scarlett- by Arthur Conan Doyle)- “It is a capital mistake to theorize before one has data”. Do you agree?
Please provide your feedback and feel free to reach out to us.
S&P Global Partners with Ness Digital Engineering to Develop New Global Talent Center in India
The Global Talent Center to become S&P Global Center of Excellence for Technology Talent
Hyderabad, India – July 17, 2017 – Ness Digital Engineering is partnering with S&P Global, a leading provider of ratings, benchmarks, analytics and data to capital and commodity markets worldwide, to cultivate its global center of excellence for technology talent in Hyderabad, India. With this collaboration, Ness Digital Engineering, a provider of digital transformation and custom software engineering services, will help S&P Global develop and evolve products and platforms that meet emerging technology opportunities across its varied business segments worldwide, including S&P Global Ratings, S&P Dow Jones Indices, and S&P Global Platts. This will add to the robust presence S&P Global’s Market Intelligence division and CRISIL already have in India.
“The Global Talent Center is an exciting opportunity for S&P Global to harness the proven technical talent present in Hyderabad thanks to its highly regarded universities and talented engineers within the region,” said Krishna Nathan, CIO at S&P Global. “The Center will enable us to accelerate digital transformation and stimulate innovation by building our knowledge assets, improving operational efficiencies, and growing our engineering team in a transformative way.”
“For over 10 years, Ness Digital Engineering has partnered closely with S&P Global to help develop its industry-leading solutions,” said Paul Lombardo, Ness Digital Engineering CEO. “We’re thrilled to further grow that relationship and help the company take advantage of the abundant talent in India to establish a global center that delivers new solutions for today’s digital economy, while also providing engineers in Hyderabad with an opportunity to become part of a team committed to innovation.”
About Ness Digital Engineering
Ness Digital Engineering designs and builds digital platforms and software that help organizations engage customers, differentiate their brands, and drive revenue growth. Our customer experience designers, software engineers and data experts partner with clients to develop roadmaps that identify ongoing opportunities to increase the value of their digital products and services. Through agile development of minimum viable products (MVPs), our clients can test new ideas in the market and continually adapt to changing business conditions—giving our clients the leverage to lead market disruption in their industries and compete more effectively to drive revenue growth. For more information, visit ness.com.
About S&P Global
S&P Global is a leading provider of transparent and independent ratings, benchmarks, analytics and data to the capital and commodity markets worldwide. The Company’s divisions include S&P Global Ratings, S&P Global Market Intelligence, S&P Dow Jones Indices and S&P Global Platts. S&P Global has approximately 20,000 employees in 31 countries. For more information, visit www.spglobal.com.
Global Manager – Corporate Communications
Ness Digital Engineering
Amazon, Whole Foods and the future of shopping
In a new article from the Internet Retailer, Moshe Kranc, CTO of Ness Digital Engineering, provides an interesting analysis of how Amazon’s decision to buy Whole Foods could be a significant move for the retailer, and also a wake-up call for its online competitors. “Whole Foods provides Amazon with an entry into high-touch experiential shopping. It also provides a laboratory where Amazon can collect data about this kind of shopping, and perhaps gain insights into how to make online shopping more exploratory and engaging. If you are a competitor in the supermarket segment, Amazon’s purchase of Whole Foods serves as a wake-up call,” Moshe notes.
DevOps Lessons Learned: Advice for IT Leaders
In a new article from the Enterprisers Project, Amit Gupta, leader of Ness Digital Engineering’s DevOps practice, provides insights about five key components to account for when establishing DevOps while stressing the importance of peeling back each layer before deploying new tools and technologies. “If we properly dissect what goes into effectively establishing DevOps, there are five layers: Collaborative culture; efficient agile development process; strong engineering practices and automation; continuous delivery capabilities, and security,” notes Amit.
Sensor Fusion Algorithms For Autonomous Driving: Part 1 — The Kalman Filter and Extended Kalman Filter
Tracking of stationary and moving objects is a critical function of Autonomous driving technologies. Signals from several sensors, including camera, radar and lidar (Light Detection and Ranging device based on pulsed laser) sensors are combined to estimate the position, velocity, trajectory and class of objects i.e. other vehicles and pedestrians. A good introduction to this topic can be found at: http://www.onenewspage.com/video/20161006/5695999/Mercedes-Benz-presents-the-Sensor-Fusion-at-2016.htm
One may question — why do we need several sensors? This is because, each sensor provides different types of information about the tracked object position with differing acuracies especially in different weather conditions. For e.g. a lidar based sensor can provide good resolution about the position but can suffer for accuracy in poor weather. On the other hand, the spatial resolution of a radar sensor is relatively poor compared to laser but provides better accuracy in poor weather. Also, unlike a lidar sensor, a radar can provide information about the velocity and bearing of the object. Laser data is also more computationally intensive because a laser sends lots of data about each individual laser point of range data, which you have to make sense of in your algorithm. The techniques used to merge information from different sensor is called senssor fusion. For reasons discussed earlier, algorithms used in sensor fusion have to deal with temporal, noisy input and generate a probabilistically sound estimate of kinematic state. This blog post covers one of the most common algorithms used in position and tracking estimation called the Kalman filter and its variation called ‘Extended Kalman Filter’. In future articles we will cover other techniques such as Unscented Kalman Filters and Particle filters.
1. The Basic Kalman Filter — using Lidar Data
The Kalman filter is over 50 years old, but is still one of the most powerful sensor fusion algorithms for smoothing noisy input data and estimating state. It assumes that location variables are gaussian i.e. can be completely parametrized by the mean and the covariance: X∼N(μ, σ²)
As information from the sensor flows, the kalman filter uses a series of state prediction and measurement update steps to update its belief about the state of the tracked object. These predict and update steps are described below.
We will use a simplified linear state space model (see https://uk.mathworks.com/help/ident/ug/what-are-state-space-models.html) to illustrate the workings of the filter. The linear state state of a system at a time t can be estimated from state at time t-1 according to the equation(1):
The next part of the Kalman filter algorithm is to use real measurements z to update the predicted state x′ by a scaling factor (called the Kalman Gain) proportional to the error between the measurment and the the predicted state.
You can find the derivation of the measurement update equations at: http://web.mit.edu/kirtley/kirtley/binlustuff/literature/control/Kalman%20filter.pdf
Enough of theory! Let’s try some code to illustrate the basic workings of the KF. Here, we simulate an object whose state is modeled as a 4-dimensional vector x=[px py vx vy] In our case, the measurement sensor is laser sensor that returns the position data but no velocity information. In order to observe velocity we need to use a Radar sensor. This will be covered in the next section when we discuss Extended Kalman filters. We will start with a set of assumptions:
The basic code for the Kalman filter steps is listed below.
"""prediction step""" def predict(x, P): x = (F * x) + u P = F * P * F.transpose() #Acceleration noise Q is assumed to be zero return x, P """measurement update step""" def update(x, P,z): # measurement update Z = matrix([z]) y = Z.transpose() - (H * x) S = H * P * H.transpose() + R K = P * H.transpose() * S.inverse() x = x + (K * y) P = (I - (K * H)) * P return x, P
The final step iterates through the measurements and applies the prediction and update steps of the filter as listed above.
plot_position_variance(x,P,edgecolor='r') #plot initial position and covariance in red for z in measurements: x,P = predict(x, P) x,P = update(x, P,z) plot_position_variance(x,P,edgecolor='b') #plot updates in blue print(x) print(P)
The above figure illustrates each iteration of the kalman filter for the px and py dimensions of the state vector along with the positional covariance. The red circle is a visualisation of our initial process uncertainty. As we go through the incremental predictions and measurement updates, we begin to develop a better estimate of state with less uncertainty (variance). As you can see, the final state vector x=[11.99, 2.05] is very close to the final measurement value and the positional state variance is also minimal at 0.05
2. The Extended Kalman filter — using Radar Data
Radar data poses a slightly more difficult challenge. Radar data is returned in Polar co-ordinates. Radar data consists of 3 components i.e.
– ρ or Range (distance from the origin)
– ϕ or bearing (the angle between ρ and x), and
– ρ˙which is the range rate.
As there is no H matrix that will map the state vector into the radar measurement space, we need a new function h(x) that will map the state space into the measurement space for the measurement update step. This function is derived by mapping the polar cordinates into the cartesian space and is defined as:
This mapping introduces a non-linearlity which would invalidate the assumptions of the kalman filter that the process and measurement models are Gaussian. The extended kalman filter approximates the nonlinear model by a local linear model and then applies the Kalman filter to this approximation. This local linear approximation is obtained by computing a first order Taylor expansion around the current estimate. The first order approximations are also called the Jacobian Matrix. The derivations of the Jacoboians are a bit involved and we will not be covering these here. However, these are well documented on several internet resources on the topic, but if you want to use these straight off the cuff then you can refer to the implementation code in the github reference below:
You can find the code for a C++ impementation of the Kalman filter in the github repository: https://github.com/asterixds/ExtendedKalmanFilter
So far we have covered some of the fundamental algorithms used in sensor fusion for object tracking. In the next part of this blog post we will look at the Unscented Kalman filter which overcomes the need to use an approximation for the projection. We will also look at a more recent and increasingly popular technique called Particle filters based on Monte Carlo Integration.
Ness Digital Engineering works with leading automotive and data technology companies in the areas of automotive safety product engineering, large scale fleet data solutions, advanced driver assistance solutions and location based services. Contact us to learn more.