Join our webinar - "Harness Automation to Transform Your IVR Testing" on May 30, 2024 | Register now.

Empowering Renewable Energy: End-to-End Solution Architecture for Wind Turbine Monitoring with AWS IoT and ML at Edge

by Noor Alsabahi, Lead Data Engineer at Ness Digital Engineering

The rapid growth of renewable energy sources has brought about a fundamental shift in our approach to power generation. Among these sources, wind turbines have emerged as a reliable and sustainable means of harnessing wind power to generate electricity. The benefits of wind energy are vast, offering clean power generation, reduced carbon emissions, and increased energy independence.

However, harnessing the potential of wind energy comes with its own set of challenges. Wind turbines operate in dynamic and ever-changing environments, making it crucial to monitor and control their performance to maximize energy production efficiently. Real-time insights into turbine health, performance, and environmental conditions are vital for ensuring optimal efficiency, predictive maintenance, and effective decision-making.

This is where AWS IoT (Internet of Things) and ML at Edge solutions come into play. In this post, we’ll share how, at Ness Digital Engineering, we leveraged the power of cloud computing, IoT connectivity, and machine learning algorithms to provide an end-to-end solution architecture that addresses the challenges faced in monitoring and controlling wind turbines.

To implement this solution, we have designed a robust solution architecture that revolves around equipping wind turbines with sensor devices, specifically Raspberry Pi sensors, through AWS IoT. These sensors capture real-time data on various parameters such as wind speed, orientation, and movement, utilizing accelerometer and gyroscope metrics. This data is securely transmitted to the cloud, processed, analyzed, and used to retrain and optimize the machine learning model over time. The deployed model on the edge device is an Autoencoder model for anomaly detection.

(figure 1: Solution Architecture)

Cloud Components:

The solution comprises two main pipelines or workflows:

  • MLOps Workflow: When there is an update on Amazon CodeCommit or for testing purposes, Amazon CodeBuild triggers the MLOps pipeline. This pipeline includes the preprocessing, training, and evaluation phases. If the evaluation surpasses the defined threshold, the model is registered in the Amazon Model Register. Model approval can be automated, but in our scenario, it is currently done manually.
  • Model Deployment Workflow: Changing the status of the SageMaker model register raises an event in EventBridge. EventBridge, in turn, triggers another Amazon CodeBuild that runs the Model Deployment workflow. This workflow involves loading the deployment code from CodeCommit, obtaining the latest model version, compiling and optimizing the model using SageMaker Neo, creating and storing Greengrass components in the component registry, and finally deploying the model to the edge device through Greengrass deployment executed in IoT Core. The IoT Core also includes a device shadow.

(Figure 2: MLOps and model deployment workflows)

Edge Device:

On the Greengrass edge device where the model is deployed, the following components are installed: Neo Model Agent, Shadow Manager, and a custom turbine component.
The turbine component directly communicates with the sensors and performs three main tasks:

  • Accumulates the data received from the sensor until a specific threshold is reached, triggering an inference request to the model.
  • Sends telemetry and inference results to the cloud via an IoT rule, forwarding the data to the Timestream database.
  • Communicates with the Shadow Manager to reflect the shadow device status on the edge device.

(Figure 3: Greengrass core device)

Data Insights and Visualization:

For data insights and visualization, Grafana is utilized and connected to the Timestream DB, allowing real-time data visualization and the generation of alerts in case of detected anomalies.

(Figure 4: Grafana real-time visualization – anomaly detected)

(Figure 5: Grafana custom data query)

Deployment Process:

The solution is implemented using CloudFormation templates. AWS CDK in .NET and AWS CLI execute the necessary scripts and CloudFormation templates on the cloud. These include various steps for setting up IoT, Greengrass Stack CloudFormation, certificates, Timestream Stack CloudFormation, manual provisioning of Raspberry Pi sensors, initial Greengrass Deployment script, Grafana script, MLOps pipeline source S3 CloudFormation, Python pipeline source code script, MLOps CloudFormation, Model Deployment CloudFormation, and EventBridge CloudFormation.

(Figure 6: Deployment)

Through this comprehensive solution, we could monitor real-time data from wind turbine sensors, detect anomalies, and control the turbine based on the insights gained. Additionally, other components and services can be easily integrated into the process for different purposes. For example, IoT SiteWise can provide an additional monitoring dashboard for operational purposes, and SNS can be linked to IoT events to capture and alert operations.
The solution architecture can also be generalized to work across multiple accounts on a large scale.

#iot #aws #greengrass #windTurbine #internetOfThings #grafana #mlops #modelDeployment #sagemaker #mlpipeline #iotcore #greenTechnology

Unlocking The Urban Mobility Revolution – How Technology Can Drive Seamless Urban Mobility

Urban mobility is on the verge of transformation, with new modes of transport, such as electric vehicles, autonomous vehicles, ridesharing, and carpooling, witnessing high adoption rates. With evolving travel patterns and behavior, urban city dwellers are spearheading the change toward sustainable mobility infrastructure, affecting everyone from OEMs to ride-share providers. Mckinsey predicts that by 2035, the share of passenger miles in private cars will drop by 15%, while new modes of travel will increase their allocation to 8%. Private car sales are expected to drop by 20% in the European Union and 30% in the United States compared to 2015. Given this disruption of traditional models, mobility players need to develop a solid future strategy to pivot and scale their operations.

Many cities and DOTs (US Department of Transportation) are working on integrating buses, trams, subways, and even bikes and scooters to create a smart city with a seamless multimodal transit network and an integrated payment system. Sustainable infrastructure such as bike lanes, pedestrian zones, and electric charging stations are becoming a regular feature in modern urban planning, revisiting the existing regulations and policies to accommodate new mobility options.

Cities are implementing data management systems for traffic management, infrastructure investments, and service optimizations as data collection and analysis become integral to urban mobility management. Data sharing between public and private transit providers is becoming critical in creating a smart transportation system.

Mobility-as-a-Service plays a pivotal role in accelerating the V2X journey by fostering connectivity, data exchange, and the development of a more integrated and intelligent transportation ecosystem. An effective MaaS framework ensures a seamless journey by providing commuters with a single app through which the trip is arranged and paid. Technologies such as artificial intelligence, deep learning, and dynamic routing play a key role in driving these innovative mobility solutions, resulting in an intelligent, seamless, and environmentally friendly ecosystem.

While commuters rarely think about the software powering a complex multi-modal transit ecosystem, there are years dedicated on the back end to deliver a flexible and seamless commute experience. Being location-dependent, MaaS is driven by complex back-end systems processing payments to ensure operators fully control their transit network while adhering to complex business rules.

Ness has been at the forefront of digital transformation for many public transit operators around the globe, leveraging its deep expertise in the industry to help public transit operators integrate with mobility-as-a-service providers in major cities worldwide. We understand how modernizing the legacy infrastructure creates seamless digital experiences, improving the everyday lives of commuters worldwide. With deep expertise in cloud-native technologies (AWS, Azure) and GenAI, we help our clients manage data effectively and save costs.

In our recently concluded webinar on “Unlocking Urban Mobility: The Mobility as a Service (MaaS) Revolution,” we discussed the changing landscape of urban mobility and how technology and innovation enable seamless urban mobility. Our expert speakers, Satish Rajaram, and Karol Grulling, shared two use cases on transforming urban mobility in large metropolitan areas and the key to successfully implementing Mobility-as-a-Service (MaaS).


We supported a local transit operator to integrate pay for parking, rent scooters, electric bikes, and taxis, and purchase tickets and passes for public transport in a single app. We built an API to integrate with the transit operator’s ticketing back office and a mobile SDK component to build UI and UX for the commuter, ensuring the transit operator manages the integration seamlessly.


We recently worked on transforming a pure analog experience into a seamless digital experience, integrating a local transit scheme called Rabbit Card into a super-app (social network + e-wallet) widely used in Thailand. Users could now top up their card through their super-app instead of physically visiting the ticket office.

We rerouted all the requests to our distributed, event-driven, cloud-native account-based ticketing system integrated with a high-performing Gate API sustaining simultaneous 2,000 gate connections. This accurately calculates how much should be paid for a particular journey. We built a complex reporting layer for real-time accurate information for the card issuer and transit operator. We also integrated with the social network to notify when a journey is completed and payment is made. This ensures a seamless journey and payment system for end-users, making public transportation convenient.

You can watch the complete webinar here.

Unlocking the urban mobility revolution through technology is crucial for addressing growing urbanization, traffic congestion, pollution, and limited transportation options. While technology plays a pivotal role, adopting a holistic approach that combines various technological solutions, data-driven insights, and collaborative efforts among stakeholders is essential.

Contact us to learn more about our manufacturing and transportation industry offerings.

Building Agile Architecture with Real-Time Data Processing

Data, data everywhere! The floodgates of data have inundated organizations, requiring systems to analyze and organize financial transactions, IoT readings, customer interactions, etc., in real-time. A modern real-time data processing architecture ingests, stores, processes, and analyzes high volumes of data from various sources in real time to improve customer experience and profitability.

A modern data processing infrastructure is hyper-scalable, handling and processing data as it’s generated, enabling complex, powerful, and real-time analytics for real-time event data streams. This architecture consists of the following key elements:

  • Ingestion Layer: This layer is an entry point for incoming data from various sources, such as sensors, APIs, IoT devices, web applications, social media feeds, databases, etc., ensuring it is captured and directed into the processing pipeline.
  • Collection Layer: This layer accommodates varying data volumes, facilitating a smooth transition between data sources and processing.
  • Processing Layer: The data processing layer is where the magic happens, with real-time data transformation and analysis. Data undergoes intricate operations such as complex event handling, filtering, aggregation, and pattern recognition to derive meaningful insights from the incoming data. Data records are processed in the order they are produced, allowing for real-time analytics and event-driven applications.
  • Storage and Aggregation Layer: Enriched data from the processing layer is stored and organized in the storage and aggregation Layer, providing scalable and cost-effective components to store streaming data. This layer includes various data storage solutions, such as time-series databases and columnar stores, and striking the right balance between real-time and historical data storage is essential for efficiency and performance.
  • Visualization Layer: The final step in real-time data processing is to leverage comprehensive monitoring and logging tools to prepare charts, reports, or graphs and provide actionable insights. Real-time dashboards and alerting mechanisms help track the health and performance of your system, detect issues, optimize performance, and ensure data quality.

A modern streaming architecture ensures flexibility to support a diverse set of use cases and provides tremendous insights from data sets that are processed in real-time. Organizations benefit dramatically from real-time data processing, as the insights enhance operations, boost monitoring and visibility for IT architecture, optimize business outcomes, and even improve overall customer experiences.

Ness leads the way in real-time data processing applications, especially in financial services, with its Risk on Cloud with Streaming (ROCS) offering. ROCS is a data-first modernization approach that leverages streaming architectures on the cloud to produce near-real-time calculations and visualizations. In fact, we modernized the risk and margining system for the world’s largest clearing corporation, Options Clearing House (OCC), from a batch-based overnight process to a near-time, event-based system.

A well-designed real-time data processing architecture is key in unlocking the full potential of data, empowering organizations to make quick decisions, proactively respond to events, and create personalized customer experiences. Learn how Ness can help you create and leverage a real-time data processing architecture for your organization.