Artificial Intelligence in Cybersecurity
The Shift We Can’t Ignore The threat landscape didn’t change overnight, but the cumulative effect has been dramatic. Cloud-native infrastructure, remote work, and interconnected supply chains have expanded the attack surface to a point where traditional perimeter thinking no longer applies. Meanwhile, attackers have professionalised – running automated campaigns, adapting mid-attack, and exploiting vulnerabilities faster than most security teams can respond. Static, rules-based defences were built for a different era. They perform adequately against known, catalogued threats. But the moment an attacker does something outside the playbook – a novel exploit, a slow-burn insider threat, a campaign that deliberately avoids triggering signatures – those systems have very little to offer. Security teams are left sifting through thousands of false positives while genuinely dangerous activity slips past. AI isn’t a fashionable upgrade to that model. It’s increasingly the only viable response to threats that outpace human reaction times. The shift from reactive defence to predictive, adaptive security isn’t a trend worth watching – it’s already underway. Why Traditional Security Models Are Falling Short Most legacy security systems are built around a simple premise: define what a threat looks like, and block anything that matches. For decades, that was enough. Threats were relatively predictable, attack surfaces were bounded, and the volume of data passing through any given network was manageable. None of those conditions hold today. The four failure modes that matter most are: Attack patterns evolve faster than signature libraries can be updated. By the time a new threat is identified, documented, and pushed as a rule update, the attacker has often already moved on. Zero-day exploits, by definition, have no existing signature. Rules-based systems are blind to them until after the damage is done. The sheer volume of data generated by modern infrastructure – logs, network flows, endpoint telemetry – makes real-time analysis impossible for human teams working with traditional tools. Alert fatigue is a genuine operational problem. When a system generates thousands of false positives a day, analysts start tuning things out. That’s exactly when real threats get missed. In sectors where the cost of a late detection is catastrophic – banking, healthcare, critical infrastructure – this isn’t a theoretical concern. Static defence cannot handle dynamic threats, and the gap between attacker speed and defender capability has been widening for years. How AI is Redefining Cybersecurity The core value of AI in a security context isn’t that it’s smarter than human analysts. It’s that it can process and correlate data at a scale that humans simply cannot, and do so continuously without fatigue. Three capabilities drive most of the practical value: Learning from data: Rather than relying on hardcoded rules, models can identify attack patterns that no analyst would have thought to encode as a signature. Anomaly detection: Identifying activity that falls outside normal behaviour, which is what makes AI effective against insider threats and novel attack techniques that leave no known signature. Near real-time response: Where a human team might take hours to triage and escalate an incident, an AI-driven system can flag, correlate, and initiate a response in seconds. These capabilities make AI particularly well-suited to the threat categories that legacy systems struggle with most: unknown exploits, slow-moving persistent threats, and attacks that deliberately mimic normal behaviour to avoid detection. That said, AI introduces its own set of challenges – around model validation, explainability, and governance – that any serious implementation needs to account for. It is not a plug-and-play solution. Core AI Techniques (With a Practical Lens) Understanding these techniques isn’t just about knowing what they do. It’s about knowing where they can fail, and what that means for testing and validation. Machine Learning (ML) Used for classification and prediction, but heavily dependent on training data quality. Testing challenge: Bias, overfitting, and model drift. Deep Learning Effective for complex threat detection (e.g., malware patterns). Testing challenge: Lack of explainability. Natural Language Processing (NLP) Used in phishing detection and threat intelligence parsing. Testing challenge: Context misinterpretation. Anomaly Detection Critical for zero-day attack detection. Testing challenge: High false positives if the baseline is weak. Where AI Actually Delivers Value Across the industry, AI has proven most effective in areas that share a common characteristic: high-volume, pattern-heavy tasks where scale is the limiting factor for human analysts. The key domains are: Threat detection and triage: faster identification and prioritisation of genuine incidents amid the noise. Endpoint security: behaviour-based protection that catches threats even when they don’t match any known signature. Phishing detection: context-aware filtering that goes beyond simple keyword matching. Network security: pattern recognition at a scale that makes human-only analysis impractical. Adaptive authentication: risk-based access control that adjusts in real time based on assessed threat level. But deployment is only part of the picture. AI security tools are not like traditional software, where you test a specific function and get a deterministic pass or fail. They behave probabilistically. Performance can degrade silently as conditions change, and the same input doesn’t always produce the same output. This changes the testing strategy fundamentally. Before deploying any AI security tool, the right questions to ask are: What is the acceptable false positive rate, and how was it measured in conditions that reflect your actual environment? How does the model perform against adversarial inputs – attacks specifically designed to evade detection? How is model drift monitored, and what triggers a retraining cycle? Can the model’s decisions be explained in enough detail for an analyst to act on them without blind trust? Organisations that treat AI as a procurement decision rather than an ongoing operational commitment tend to get disappointing results. The technology requires sustained attention to perform reliably. Conclusion AI is not replacing cybersecurity professionals. It’s changing what they spend their time on. The manual, high-volume work of correlating logs and triaging alerts is increasingly something machines handle better. The judgement calls, the contextual decisions, and the communication with stakeholders – those remain firmly human responsibilities.
AI is the future of test automation- Are you Ready?
Traditional QE Isn’t Working Anymore Traditional tried-and-tested methods of testing and quality need to catch up in today’s changing environment. A siloed approach: Typical QE teams are separated from development teams. This structure concentrates on optimizing the subcomponents and deviates from the true purpose of enhancing the user experience. Slowing overall engineering velocity: Traditional quality engineering has been chiefly manual, impeding rapid development and operations procedures. Expensive: Traditional QE requires significant engineering resources and costs 30%–40% of the overall expenditure. An afterthought: For decades, the testing strategy has been put off until the end of a product cycle, which is too late and can cause release delays and budget overruns. THE NEW DIGITAL ERA REQUIRES INCREASED SPEED & AGILITY DevOps and intelligent automation, as well as the proliferation of digital applications, have considerably challenged traditional techniques for application testing in recent years. Delivery times have gone from months to weeks, and nowadays, testing has moved to the left and right of the software development lifecycle. Agile and DevOps have combined development and testing into a single, continuous process. Quality engineering has changed from testing to starting with the planning of the first application. It creates a constant feedback loop that lets you plan for the unexpected and act on it. However, to properly comprehend the magnitude of the evolution from testing to quality engineering, we should first recognize how data has impacted software development. Data can do more than just power automation use cases and AI learning datasets for repetitive development and testing processes. The enormous amounts of data users create daily to make it more important for quality engineers to predict risk, find opportunities, increase speed and agility, and reduce technical debt. Quality engineering is changing in tandem with the ever-increasing cyber security concerns. Today’s quality engineering role must enable faster application, product, and service delivery and act as an enabler, not a barrier, to digital transformation. TAKING QE IN THE NEW As these changes in quality, technology, people, and organizations take hold, QE will grow into a role that is more pervasive, real-time, and based on insights. AI-led autonomous frameworks will support this to make sure business continuity and value. Testing will evolve away from traditional ways and toward new ideas and methodologies appropriate for the application engineering world of the future across five dimensions: data, frameworks, process, technology, and organization. FROM APPLICATION-FOCUSED TO PURPOSE-DRIVEN Today’s rapid growth of enterprise application testing environments shows no signs of slowing. As it evolves, QE’s focus on apps will be less defined by its alignment with business objectives. This means testing, monitoring, and making real-time fixes to ensure that the business “works” as well as the code. It also entails creating self-learning, self-adapting systems assisted by machine learning and advanced analytics. Tavant is actively planning for this future. We are propelling QE into the future with breakthroughs in holistic QE strategy, incorporated cognitive and machine learning capabilities, and end-to-end automation. This changes everything, from test planning and test case development to test execution and environment setup, and it helps QE reimagine its role in the future enterprise. Tavant Quality Engineering Services helps organizations engineer quality into their processes by incorporating a whole gamut of services, tools, and techniques to elevate the end-user experience. AI-Powered Next-Gen Test Automation Framework Tavant’s AI-powered next-gen test automation platform, FIRE (Framework for Intelligent and Rapid Execution), is Tavant’s proprietary suite of solution accelerators aimed at optimizing the overall testing effort and delivering a high-quality product. It is a comprehensive tool and technology agnostic-test automation framework. This framework can orchestrate multiple automation tools and technologies, including (but not limited to) Selenium, Appium, Cypress, Protractor, Microfocus and Java, C#, F#, Python, and PHP. The test automation framework ensures speed to market and superior quality software. FIRE 5.0 accelerates the time to market while simultaneously aiding developers to enable a dual shift of the software development lifecycle to gauge the consumer experience and provide continuous feedback into the system. It offers comprehensive test automation coverage and efficiency of more than 90%. Importance of Quality Assurance & Testing in the Finance Industry The financial industry is on the verge of transformation. Mobile banking, investment, insurance, app payments, advances in technologies such as cloud, mobile, AI, agile, and DevOps, and concerns about fraud detection management, data visualization analytics, risk and compliance management, and digital lending require specialized testing. Tavant’s Quality Engineering Services have been designed with the financial industry in mind. Our quality engineering services can help you with an error-free application and accelerate your time to market at a lower cost. Visit here to learn more.
Tavant Partners with One of the Largest Commercial Truck Manufacturers in the world to Provide a Connected and Seamless Service Experience
SANTA CLARA, Calif, and EINDHOVEN, Netherlands, NOVEMBER 1 2022 SANTA CLARA, Calif, and EINDHOVEN, Netherlands, NOVEMBER 1 2022 – Tavant, a digital products and solutions company and a global leader in service life-cycle management, today announced that it has partnered with Daimler Truck AG (DTAG), one of the world’s largest commercial vehicle manufacturers, to provide warranty and claim management solutions for its European brands. “Upon deployment, Tavant Warranty will enable seamless integration with all backend and frontend systems, bring in workflow automation, better configurability, and amplify productivity. We are excited to be working with Daimler Truck. This alliance opens the pathway to great synergy for accelerating the pace of service innovation,” said Vikas Khosla, Chief Revenue Officer, Hitech, Tavant. Leader in IDC MarketScape for warranty and service contract management, Tavant Warranty is a next-generation warranty management software solution. It uses AI and machine learning capabilities that transform the warranty and service process by constantly adapting to an analytics-first decision-making engine. The solution enables manufacturers to make intelligent warranty decisions, allowing businesses to concentrate on more complex and significant value-driven problem statements. “This collaboration emphasizes our commitment to digital growth. Tavant’s connected service life-cycle solutions integrate seamlessly with legacy systems to harmonize business processes. Throughout this transformation journey, we will implement a modern, future-ready, and integrated warranty and goodwill system, provide a 360 view of the service life-cycle, and a superior aftersales experience to their customers,” said Roshan Pinto, Head of Manufacturing, Tavant. Find Tavant on LinkedIn and Twitter.
Why Cloud and Data Analytics go hand in hand?
As per Gartner, adoption of data and analytics will increase from 35% to 50% in 2023, driven by industry vertical and domain-specific augmented analytics solutions. By 2024, 75% of organizations will have deployed multiple data hubs to drive mission-critical data and analytics sharing and governance. The research also highlights that nearly 70% of enterprises will use cloud and cloud-based AI infrastructure to operationalize AI systems for their businesses over the next two years. Cloud adoption has significantly accelerated post-pandemic, with enterprises increasingly focusing on the digital transformation across their business functions. One of the critical drivers in cloud adoption is the onset of data-driven strategy across industries. Cloud has helped in the paradigm shift to implementing data and analytics solutions and fast-tracked the time to market for data analytics solutions. A recent survey by IDG Research and Tavant indicates that data analytics is a key focus area for organizations across industry verticals in the USA, where enterprises are looking at leveraging the cloud to implement data-driven systems. More than 80% of C-level survey respondents plan to leverage the cloud to drive enterprise data analytics. Thus, it is evident that cloud technology is pivotal in driving faster data analytics adoption, the emergence of next-gen SaaS products, and modern-day cCloud data platforms. Role of Cloud in Data Analytics: With the wide range of solutions focused on infrastructure and data analytics-specific services, the cloud has acted as a catalyst in driving the adoption of Data analytics. Today, Cloud Service Providers (CSPs) are accelerating the data analytics adoption with broadly two service offerings; Infrastructure services – The fundamental cloud Compute and Storage solutions help organizations implement custom solutions faster and address scalability challenges. The mere availability of computing and storage faster elasticity has enabled enterprises to adapt quickly. Modern data platforms also leveraged infrastructure services in providing cloud-agnostic services. Data Analytics services – CSPs are leading cloud providers to provide data-specific services to build Cloud-native data solutions. Examples are Hadoop on Cloud as PAAS – AWS EMR, Azure HDInsight, GCP Dataproc, and the related services to create a complete data solution. The CSPs will glue these cloud components together to build custom solutions for future enterprises. Cloud-enabled Data Analytics solutions As organizations embark on building complex data solutions, the cloud becomes an integral component of the data architecture. Understanding the various alternatives helps select the right technology based on business context. Below is the broad category of cloud-driven solutions. Cloud infrastructure-focused data solution The solution leverages cloud infrastructure services to deploy data analytics solutions faster. These solutions are most suited for companies that need to rehost existing data solutions from an on-premises environment to the cloud or build a custom solution from scratch. Examples include AWS S3/Azure ADLS/GCP cloud storage as the data lake and various computing services by AWS/Azure/GCP Cloud-specific data solutions These solutions leverage cloud-native data services to build data analytics solutions. The data services and pre-built integrations across different cloud services are helpful for enterprises and CSPs to co-create custom solutions faster with data privacy and security needs. Examples include EMR, Kinesis, S3, from AWS, HDInsight, ADLS, NoSQL databases, Stream Analytics from Azure, and Dataproc, Storage, NoSQL DBs, Pub/Sub from GCP. Cloud Datawarehouse Cloud-native Datawarehouse solutions by CSPs help to deploy enterprise-grade Datawarehouse faster. Examples include AWS Redshift, GCP BigQuery, and Azure Synapse analytics, which have pParallel processing, pre-built integrations for ingesting data, and AI/ML capabilities. Modern data platform on cloud -Modern Cloud-native data platforms like Databricks and Snowflake focus on building a single platform addressing the needs of Data Analytics. As cloud and data analytics drive the adoption of each other, it is imperative to understand the mutual dependence and leverage it while planning for cloud adoption or data analytics solutions within the organizations.
Tavant Announces Significant Expansion Across Europe
Eindhoven, Netherlands, June 24, 2022 Eindhoven, Netherlands, June 24, 2022 – Tavant, Silicon Valley’s leading digital products and solutions company, today announced its expansion in the European market to serve its growing customer base. This will involve a major expansion of its current operations in the Netherlands (Eindhoven) and the UK (Nottingham), as well as the opening of a new near-shore development center in Portugal. This expansion will be driven by local hiring, training of local talent, as well as looking at the acquisition of a Europe based systems integrator. This will empower its customers across the UK, Netherlands, Belgium, Spain, Norway, France, Switzerland, Germany, and Italy to further grow the digital economy. “We are very pleased to welcome Tavant’s further expansion to the Eindhoven region. The AI & Data Technologies are slowly becoming part of our everyday lives and we see it touching many parts of industry in Brabant, from manufacturing to logistics, field service to customer delivery. Inspection, picking, and autonomous navigation are just a few areas where we see great, real-world applications and many more opportunities for Tavant,” said Guido Leestemaker, Project Manager, Foreign Investments at the Brabant Development Agency. “It’s our privilege to work with our customers across Europe – a region that continues to bring innovations to market every day. As our world begins to come together in person once again, our Europe expansion is all about closer collaboration with our partners and customers. The decision to expand our operations in Europe is not only a logical step in developing our business but, more importantly, is a response to the growing demand of our European clients who praise our speed, agility, adaptability, and our ability to turn their business goals into meaningful business outcomes. We would also like to extend our thanks to the Netherlands Foreign Investment Agency (NFIA) and the Brabant Development Agency (BOM) for their support in this expansion in the Netherlands,” said Sarvesh Mahesh, CEO, Tavant. “This is a perfect next step as we evolve our business as the leading product and solutions company across EMEA and globally. We are purpose-built to help businesses adopt next-generation digital technologies and are enormously proud of our track record of delivering great client outcomes,” Mahesh substantiated. The European expansion is the latest in a series of major milestones for Tavant in recent years. Tavant also recently opened its second U.S. development and innovation center in Dallas, TX, and added 300 new employees across the U.S. Over the last few years, Tavant has increased its focus on product and platform development, leveraging connected technologies such as IoT, machine learning, data science, and AI & advanced analytics to collaborate with customers on their digital journey. Tavant has always been known as a top employer, which helps it in attracting and retaining top talent. More recently, Tavant’s revenue has grown by over 40% YoY, creating a foundation for Tavant’s re-investment in the European market. Find Tavant on LinkedIn and Twitter. Media Contact: Vibhor Mishra [email protected]
Transforming IoT Data into Actionable Insights with Time Series Insights
Time Series Insights for IoT data: Generally, IoT data typically consist of time series data, which makes sense when observed over a period of time, like a sensor’s behavioral change, etc. Billions of data is getting generated from IoT these days, and it needs to be stored in a repository. But it’s challenging to store this data in a way, where you want to use it in near real-time to be processed to derive meaningful insights when needed in machine-critical situations. So, we need to store this data in a way that it makes sense. This calls for a service that can scale massively and help operators find insights quickly, Azure Time Series Insights. Introduction to Azure Time Series Insights: Azure Time Series Insights is a serverless, fully managed data analytics solution (PaaS), that users can use to integrate with their constantly changing data like data from several sensors or machines, data from airlines, satellites, etc. Any data that can be generated on a large scale and needs to be analyzed can be used through Azure Time Series Insights. Azure Time Series Insights architecture: The above figure shows a high-level architecture of how Azure TSI can be implemented in a real-life scenario. Time series real time data can be generated by various sources like satellites, mobile devices, medical devices, sensors, etc. Azure IoT Hub or Event hubs can be used to fetch the data from these devices into the Azure environment. Further, this data can be processed using services such as Stream analytics, Logic apps and Azure functions and computed signals from the processing pipeline are pushed to Azure Time Series Insights for storing and analytics. Once in the Time series insights platform, the data can be used for visualization. The data can also be queried and aggregated accordingly. In additional, customers can also leverage existing analytics and machine learning capabilities on top of the data available in Time Series Insights platform. Data from Time Series insights can be further processed using Databricks and pre-trained machine learning (ML) models can be applied to offer predictions in real time. Components of Azure Time Series Insights: Integration: Time Series Insights provides easy integration for the data generated by IoT devices by allowing connection between the cloud gateways like IoT hub and Event hubs. Data from these can be easily consumed in JSON structures, cleaned and stored in columnar store. Storage: Azure TSI also takes care of the data that is to be retained in the system for querying and visualizing the data. By default, data is stored on solid state drives (SSDs) for fast retrieval and can be retained for upto 400 days. Data visualization: Another component of Azure TSI, data visualization helps data fetched from multiple data sources and stored in the columnar stores, to be visualized in the form of line charts or heat maps. Query Service: Time Series Insights also provides a query service using which you can integrate Time Series Insights into your custom applications. Conclusion: Azure Time Series Insights helps you to easily connect to billions of events in Azure IoT hub or Event hubs, visualize and analyze those events to spot the anomalies and discover hidden trends in your data. It can both store as well as visualize the data. Alternatively, one can also have the capabilities to run queries against this data and obtain more simplified results.
AI-based Attribution Models – The Future of ROI-focused advertising
The current economic climate is putting advertisers and marketers under tremendous pressure to demonstrate ROI from their marketing dollars. Gone are the Mad Men days of advertising when the art of storytelling was all it took. Today, advertisers are spending in an omnichannel environment and must account for every ad dollar. It’s no longer all about reach – or clicks on online ads. The entire focus of advertising has evolved from reach or clicks to the desired outcome – usually a purchase – either online or offline. How do marketers know which of their multitude of channels and touchpoints are performing on par and responsible for consumers’ desired actions? Enter marketing attribution As the marketing ecosystem gets more complex, the platforms that need marketing dollars to increase in number, and the demand for justifiable ROI from marketing reaches unprecedented decibels, sound measurement, and attribution have become the holy grail for brands and CMOs. They are seeing an immediate need to link different marketing touchpoints in their attribution and decision-making models to truly understand the new consumers’ purchasing path. Marketers have been following simplistic attribution models such as a first click or last click or attributing credit to the first or last touchpoint in the path to purchase. However, these attribution models are hardly comprehensive or data-driven, and they rarely say anything about the consumers’ intent. AI-based algorithms now play a significant role in building sound attribution models for the new complex, non-linear buying journeys and multitude of marketing channels, platforms, and touchpoints. Custom attribution models with AI at the core We have developed four key models for AI-based attribution that are truly data-driven and cutting-edge. AI-based custom attribution models are truly compelling in the new-age marketing ecosystem. Our four attribution methods are based on all events and channels where customer touchpoints exist and can predict – to a large degree of accuracy – whether a touchpoint led to conversion or not. What are these four methods? Here’s a quick summary: Logistical Regression – It is a well-established statistical model that takes inputs from existing touchpoint data and predicts which class the data should belong to. A non-linear function is applied to each touchpoint. Smaller the value, smaller the weightage assigned to it. Using these touchpoints, the model is trained over time. Each touchpoint becomes a variable in the logistical regression model, predicting conversion based on historical data to a reasonable degree of accuracy. Shapley Value – The Shapley Value model takes a game-theoretic approach to multi-touch attribution. The core idea is to keep or remove a channel and then ascertain the outcome. This naturally tells you your highest performing and lowest performing channels and is a fair and transparent way to attribute credit to each channel or combination of channels. Markov Chains – This model considers the sequence of the customer journey, i.e., the likelihood of each customer being exposed to some marketing tactic and the potential next step in the journey. In summary, the Markov Chains model focuses on the probability of each consumer transitioning from one exposure to the next marketing exposure and taking a desirable action in the process, such as a website visit or a purchase. The model considers all possible conversion paths. It gives appropriate weightage to each exposure on the customer’s journey to conversion. We then take away one of the channels and see the impact on conversion and subsequently ascertain the value of that channel in the attribution model. Hidden Markov Model – Hidden Markov Model, although new, is one of the most effective attribution models in marketing. It attempts to determine the state of mind of each consumer when they perform any action during the path to purchase. For example, what state of mind is the consumer in when he or she visits the website, searches, clicks on an ad, etc. This determines whether the action will lead to purchase or not. The Hidden Markov Model has had a significant impact on ML, and its impact on marketing and advertising is only beginning. It is safe to say that in the world of clicks – sometimes even inadvertent – the Hidden Markov Model can truly predict the role of each channel in bringing the consumer closer to desired actions like purchase. The Bottom Line: AI-based custom models are the future of marketing attribution. Evidently, AI-based attribution models can track each consumer at each stage of the buying journey and understand the importance of each “moment” and “action” in the customer journey. This helps advertisers truly understand the performance of each touchpoint and channel in the buying journey and optimize media spend continuously during campaigns. AI-based attribution models are driving the next generation of ROI-focused marketing. Are you ready to up your measurement game with AI? If yes, then reach out to [email protected] or visit us here to know more.
Cracking the AI Implementation Code by Operationalizing AI
AI is increasingly affecting our daily lives as more and more tools are using AI. There is no doubt that enterprises are taking a serious interest in adopting artificial intelligence and machine learning. But the knowledge of how it must be deployed to accelerate automation and transform business processes is still in its nascent stages. ” src=”https://www.tavant.com/sites/default/files/blog-image/Operationalizing%20AI%20blog.jpg” alt=”Operationalizing AI” width=”724″ height=”408″ style=”box-sizing: border-box; border: 0px; vertical-align: middle; max-width: 100%; width: auto; height: auto; float: none; padding-bottom: 35px; padding-right: 10px; padding-top: 10px; display: block; margin-left: auto; margin-right: auto;”> AI IS HERE. BUT WHO REALLY HAS IT? AI is making inroads into our lives. According to a new KPMG survey of industrial manufacturing business leaders, the next two years will see AI technology having varying impacts on industry needs: 21% for product design, development, and engineering, 21% for maintenance operations and 15% for production/assembly. In fact 61% of business leaders believe that increased productivity is the most significant potential benefit of AI adoption . Despite the interest, enterprises across industries still struggle with the process of AI implementation to achieve predicted business benefits. HURDLES TO PRACTICAL AI IMPLEMENTATION As companies race to digitize and embrace edge technologies, the transition often requires business leaders to shift their thinking from traditional software engineering expectations. There are many reasons AI and machine learning models don’t necessarily pay off. Some of these include: 1.Lack of Qualified Data Scientists Data science is an essential aspect of developing a suitable machine learning methodology. But the growth of data processing in AI has led to a demand for data scientists who can help turn raw data into business value. This shortage can be overcome by either outsourcing the ML model development or training employees already working with data in ML model programming. 2.Poor Data Quality AI and machine learning tools rely on clean data to train algorithms. And businesses that do not have control over their data quality or data management will struggle to make their AI initiatives successful. Data engineering enables enterprises to maximize the value of their data assets. By working towards enabling cleaner data sets, businesses can deploy machine learning algorithms to design accurate predictive analyses. 3.Undefined End Results What performance metrics are to be measured when developing and selecting machine learning models? Businesses often fail to know the desired level of performance before an AI project begins, leading to a mismatch between model results and expectations. Understanding the project deployment maturity levels can help leaders understand the progress needed to adopt AI successfully. 4.Difficulties in replicating ML model results Incremental data and different environments often cause ML models to perform differently. ML models need to be updated or refreshed to account for data drift, deterioration or anomalous data. Rather than upgrading the ML model every time, businesses need to create repeatable modelling processes to ensure continuous learning happens during production. FROM EXPERIMENTATION TO EXECUTION Operationalizing AI involves combining ML learning methodologies with software engineering principles to create a production-grade solution. Using established frameworks can help companies find a starting point to formulate best practices to go forward. Atul Varshneya, VP of the Artificial Intelligence Practice at Tavant, has detailed an approach and points to consider for businesses looking to operationalize their AI initiatives. If you are looking for ways to move your machine learning projects from experimentation to execution, watch this recorded webinar. Are you looking to overcome the challenges in operationalizing AI for your business? If yes, then reach out to us at [email protected]. Source: Impact of AI on industrial manufacturing (kpmg.us)
Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects
As discussed in my previous blog posts, a lot of research is being done in ad attribution and media mix modeling. Today I’ll introduce another paper that provides some interesting analysis. Fair warning, you should have a basic idea of Bayesian regression before reading this. You can find a great introduction here. Carryover and Shape Effects The authors’ most exciting contribution is incorporating carryover and shape effects in their media mix model. Carryover effects try to model the impact of media spend over a future period. Since media spend influences consumers on buying a product or service, the impact of such spending doesn’t just last for the time an advertisement is aired but for a more extended period. The authors transform the time series of media spend using a decay function for accounting for such carryover effects. They use the adstock function as described below : wm is a non-negative weight function, and the media spend effect is the weighted average of media spend of the current period and previous L-1 periods. The authors introduce two types of weight functions, geometric decay (where media spend peaks when an advertisement is aired) and delayed adstock (where the impact of media spend rises sometimes after an ad is aired). A visualization describing the effects of the weight functions can be seen below, taken from the paper. Next, the authors discuss shape effects. Shape effects aim to capture diminishing returns on media spend. For example, it is valid to assume that for a specific medium, the rate at which media spends rises is dramatic from 0 to $50 but reduces significantly from $100 to $150. The authors use a Hill function to model shape effects. The discussion of Hill functions is beyond the scope of this blog, but the regression coefficients can be multiplied by the Hill function to get the following form : The hill function for media spend is a point transformation, as opposed to earlier discussed carryover effects. The following graph, taken from the paper, gives a visual representation of diminishing returns, given different parameter values in the Hill function : Both these transformations can be applied to media spend. Depending on individual use cases, one must decide which transformation to apply first. The authors apply the adstock transformation first and then the shape transformation. The final sales at time t, which can be described as y_t , can be modeled using the following equation : To simplify, this equation models sales as a function of some baseline sales τ in addition to transformed media spend effects of control variables, and random noise. Why Bayesian Regression A common question could be: Why estimate these parameters using bayesian regression? The answer lies in the fact that Bayesian regression lets us quantify the uncertainty in our predictions, and more importantly, allows us to set priors on our parameters. For example, it is valid to assume that media spend will never have a negative effect on sales, which allows us to set informative priors on media spend coefficients (constraining them to be non-negative values). The authors then explain their implementation of this model to real-world datasets. They use Gibbs sampling to sample from their model and implement this in STAN. However, multiple techniques for sampling from the posterior distribution and their code can be replicated easily using PyMC3. Please take a look at the fundamentals of Bayesian Regression if this isn’t making much sense. The parameter estimates obtained from the model can then be plugged into a linear optimization algorithm that conditions on a fixed media spend budget to find the best media mix given a set of channels. The linear optimization algorithm introduced by the author is beyond the scope of this post, but I might discuss it in my next one. Stay tuned!
Building Trustworthy and Ethical AI is everyone’s responsibility
Whether you realized or not, Artificial Intelligence (AI) has quickly become part of our daily life. With traditional industry and businesses like fintech, media, healthcare, pharmaceuticals, and manufacturing adopting AI rapidly in recent years, concerns related to Ethics and Trustworthiness have been mounting. Today, AI ‘assists’ many critical decisions influencing people’s life and well-being, for example, creditworthiness, mortgage approval, disease diagnosis, employment fitment, and so on. It was observed that even with human oversight, complex AI systems may end up doing more societal harm than social good. Building Trustworthy and Ethical AI is a collective responsibility. We must apply fundamentals throughout the lifecycle of AI, for example, product definition, data collection, preprocessing, model tuning, post-processing, production deployment, and decommissioning phases. No doubt Government and Regulators have a role to play through monitoring and ensuring a level playing field for everyone, the same is for people building, deploying, and using AI systems. This includes executive leadership, product managers, developers, MLOps engineers, data scientists, test engineers, HR/Training teams, and users. Bias and unfairness While Trustworthy and Ethical AI is a broader topic, it’s tightly coupled with the prevention of of Bias and Unfairness. As the National Security Commission on Artificial Intelligence (NSCAI) observed in a recent report: “Left unchecked, seemingly neutral artificial intelligence (AI) tools can and will perpetuate inequalities and, in effect, automate discrimination.” AI learns from observations made on past data. It learns the features of data and simplifies data representations for the purpose of finding patterns. During this process, data gets mapped to lower-dimensional (or latent) space in which data points that are “similar” are closer together on the graph. To give an example, even if we drop an undesired feature like ‘race’ from the training data, the algorithm will still learn indirectly through latent features like zip code. This means, just dropping ‘race’ will not be enough to prevent the AI learning biases from the data. This also brings out the fact that data ‘bias’ and ‘unfairness’ reflect the truth of the society we live in. With not enough data points belonging to underrepresented sections of the society, high chances that they will be negatively impacted by AI decision-making. Moreover, AI will create more data with its ‘skewed’ learning which will be used to train it further and eventually create further disparity through its decision-making. Trustworthy and Ethical AI is important By definition, Trustworthiness means “the ability to be relied on as honest or truthful”. Organizations must ensure their AI systems are trustworthy, in absence of trust, undesired consequences may occur, including but not limited to business, reputation and goodwill loss, lawsuits, and class actions that can be potentially life-threatening for a business. On the other hand, Governments and Society must ensure that AI systems follow Ethical principles for the greater good of common citizens, one great example is UNESCO Ethical AI Recommendations. As per the European Commission Ethics Guidelines for Trustworthy AI, Trustworthy AI must be Lawful, Ethical, and Robust. Respect for human autonomy, fairness, explicability, and prevention of harm are four critical founding principles of Trustworthy AI. It’s critical that AI should work for human wellbeing, ensure safety, should be always under humans’ control, and never ever should harm any human being.. Who is driving Ethical AI? Realization of Trustworthy AI is envisioned through the following actions: Who is driving Ethical AI? Leading tech companies have already announced one or another kind of Ethical AI initiatives and governance. As there is no common ground in terms of benchmark principals, guidelines, and framework, it’s difficult to assess whether the intent is genuine or merely optics. As AI will have a profound impact on society and well being of common citizens, just ‘self-certification’ will not be enough. Governments should (some have started already) define the principles, policy, guidelines and establish an effective oversight and regulatory mechanism. This will help to ensure that common citizens are protected from intended/ unintended negative fallouts of AI. As AI evolves, frameworks and regulations should also evolve. Recently, the US Federal government signed Executive Order On Advancing Racial Equity and Support for Underserved Communities, however, more needs to be done. EU, UN & DoD have already taken the lead on this topic, with European Commission Ethics Guidelines for Trustworthy AI, UNESCO Elaboration of a Recommendation on the ethics of artificial intelligence and US Department of Defense Ethical Principles for Artificial Intelligence should be considered as baseline work towards defining a practical and mature guideline towards Trustworthy and Ethical AI. Plan of action Here we attempt to identify suggested actions for involved actors. This is in no way an all-inclusive list and should be taken as only a baseline and should be updated to support a particular case: Conclusion We all have actions to build Trustworthy and Ethical AI for the larger good of society (and humanity). With coordinated and persistent efforts, it is definitely possible.