Brace yourself for #MarTech
The AdTech industry has witnessed significant growth in recent years, but there is another industry growing rapidly and which could likely overtake AdTech in the future. MarTech, refers to the innovation in marketing technology outside the context of advertising, focused on experience and data-driven marketing. AdTech, on the other hand, refers to advertising technology focused on advertising and serving technologies such as ad-server and real-time bidding solutions. In the recent past, more and more organizations are adopting MarTech to innovate marketing pipelines, infrastructure, and workflows, to achieve higher operational efficiency, greater insights and better planning for marketing budgets. The image below, published by Scott Brinker, Editor, ChiefMartech, gives an idea about the scale and pace at which the world of Martech, is growing. Scott mentions that the number of companies adopting MarTech has doubled over the last one year with the overall number crossing 2000. Expansion of the MarTech landscape indicates the evolution of marketing, the increasing role of technology in marketing and the heterogeneous nature of MarTech field. Scott draws attention to key areas where innovation and success are clearly visible: Internet – Scott attributes the success of the digital world to the penetration and affordability of the Internet in people’s daily lives. Infrastructure – The transformation of data storage and presentation capabilities has enabled gathering and processing of data for greater understanding of consumer behavior. Scott says that most of this can be attributed to big data, cloud computing, and mobile/web app development Marketing Backbone Platforms – Platforms such as CRM, e-commerce engines, etc. have brought businesses closer to end customers even as they address pain points. Marketing Middleware – Data Management Platforms (DMPs), CDPs, and tag management software have enriched data with metrics that were not available earlier. These metrics have helped to improve operational efficiency and overall productivity for companies. Marketing experiences – Scott calls them the `front-office’ of marketing. These technologies revolve around customer lifecycle such as social media, email, and A/B testing Marketing Operations – Mostly associated with Business Intelligence, Analytics, Visualization and Data Science, marketing operations helps to interpret and find solutions To summarize, even though a lot of action is happening around MarTech, the fact remains that there is huge potential for growth. We will likely witness a shift of focus away from AdTech towards MarTech.
Gaining Intelligence from Gaming – Game Analytics Platform
A successful decades-old, global, video game developer, publisher and hardware company generates several gigabytes of telemetry data on a daily basis. This company has several successful game franchises that attract millions of players on iOS and Android devices every day, and all these clicks and interactions have resulted in incredibly valuable data. If this data can be aggregated and analyzed, it will hold the key to player engagement and retention. An example of the kind of data includes the event streams that indicate when players are playing, duration, levels reached and the money they spend on buying virtual goods such as new levels and avatars. Social networks complement this information with details about players’ real-life preferences. To analyze this data and use it for intelligence an Analytics platform had to be created. Given below are the steps used for building this solution: Solution Architecture The solution was built on Amazon’s cloud platform using Amazon Web Services (AWS). Software Development Kits (SDKs) for the different game technology platforms, including Android and iOS, were used to enable the games to push events data to the data collection server with minimal programming. Data Collection For the purpose of data collection, server using Node.js, which collects high velocity data from players’ mobile devices and writes it in real time to folders on an S3 (Amazon Simple Storage Service) bucket was used. This provides an event-driven architecture and a non-blocking I/O API that optimizes the throughput and scalability of data. An Amazon EMR cluster to process the collected event streams from S3 multiple times every day was adopted. For each batch, a cluster on demand, based on the data volume, wrote the results back to S3, and then shut down the cluster to save on costs. MapReduce jobs validated and cleaned the event data and wrote the results back to S3. Hive jobs then further processed these files to generate facts, dimensions and aggregated facts for later analysis. Data Persistence and Visualization To support rapid query and analysis, the Hive output was loaded into a data mart built on MySQL (and later experimented with Amazon RedShift as well) and Tableau to create dashboards and interactive charts were used. As a result of this solution, the game company gained valuable insights, including: The conversion rates of players from free to paying customers based on geography, game title, and other dimensions The skew in the distribution of paying customers (a small number of players accounted for a large part of the total spending) An understanding of each player’s playtime across multiple games (surfacing opportunities for cross-promotion within each game) Detection of fraud through comparisons of the game’s telemetry data about purchases with the app store data for in-app purchases (it turned out that hackers had exploited a vulnerability in the game design that was quickly corrected)
Parallel Job Execution in Pentaho with Dynamic Configuration
We had to design and build a data warehouse for multi-tenant architecture. However, there were multiple clients with different data metrics and source databases while the data model (dimensions and facts) was common. Thus, to start with, we developed the ETL (Extract, Transform & Load) jobs for a single client which had to be scaled for this new requirement. To load the data using the same ETL job, one can change the source and target database configurations in jdbc.properties file but this is not a scalable approach as the properties file needs to be modified every time a job has to be executed. Moreover, since Pentaho refers JNDI (Java Naming and Directory Interface) connections using the name, one cannot define two JNDI connections with the same name. Also, there is a good possibility of executing the same job in parallel for different source and target databases. In such situations, multiple parameters might override during parallel execution due to common kettle.properties file. An inefficient way to tackle this problem would be to create a separate job for each client. Imagine a scenario where there are hundreds of ETL jobs and tens of clients! It would lead to a huge duplication of effort and would be operationally ineffective. We, therefore, wanted to achieve our objective with minimal rework and a good design approach which would be maintainable and operationally effective. The Solution: In case one has installed PDI on a server at path “/opt/data-integration” and ran a PDI job through below kitchen command then by default it searches kettle.properties and repository.xml at path “KETTLE_HOME/.kettle” and searches jdbc.properties at path “/opt/data-integration/simple-jndi”. /opt/data-integration/kitchen.sh -rep test -job load_data_job To overcome this problem, one can assign the path while firing the kitchen command. For example: KETTLE_HOME=”/data/client1/” KETTLE_JNDI_ROOT=”/data/client1/jndi/” /opt/data-integration/kitchen.sh -rep test -job load_data_job KETTLE_HOME=”/data/client2/” KETTLE_JNDI_ROOT=”/data/client2/jndi/” /opt/data-integration/kitchen.sh -rep test -job load_data_job In the above example, we have separated kettle.properties and jdbc_properies files for both clients at a different location. If you run the first kitchen command, it searches kettle.properties at path “=”/data/client1/” and jdbc.properties file at path “=”/data/client1/jndi/”. If you run the second kitchen command, it searches kettle.properties at path “=”/data/client2/” and jdbc.properties file at path “=”/data/client2/jndi/”. Similarly, for each new client one can set up the configuration in a new location and point the kitchen command to new kettle and jdbc properties files. This gives the benefit of reusing existing ETL jobs, avoid conflict between parallel executions while it provides the flexibility to scale when required.
Data Cloning Through Pentaho Data Integration Clone Step
Splitting rows based on a column value. The input data comprises of ticket booking records defining number of seats booked at event, section and row level. It also contains the starting and last seat number. Objective It was required to split ticket blocks within event_name + section_name + row_name as follows: convert the record where num_seats > 1 into as many records as num_seats assign values as follow in split records num_seats = 1 for each record seat_num = individual seat within the block original seat_num + i where “i” is counter from 0 to num_seats – 1 last_seat = new value of the seat_num as above population logic of rest of the column remains unchanged Sample Data Input: Expected Output: So based on the above screenshots, we need to split the incoming input rows based on the num_seats field .So for first input row where num_seats=4 we need to generate 4 records as per the rules defined above. Solution: Pentaho provides a clone row step that can clone objects or rows in the same way as the main row based on a column value. Refer to the below screenshot for the solution: Table input: This step will load the input data. Clone row: This step will create the clone objects or rows similar to the main row Nr clone in field: will specify column value to be used for cloning Add clone flag to output: will put the flag=N for the original row and Flag=Y for clone rows Clone num field (seat_index_rownum): will add the index value (0,1,2,..). Filter rows: Remove the original (non-cloned) row (where clone?=N). Calculator and Select Values: Calculate the seat number and replace the original fields (num_seats, seat_num, last_seat, etc.) with the new values. Table Output: Loading the data into the target table.
The Changing Landscape of the Ad Technology World
Last year saw a significant increase in the dollars spent on digital versus traditional media advertising and this has prompted marketers to spend more money online. 2014 has also been an exciting year for mobile advertising as more and more agencies are trying to create a niche for themselves through evolving standards, benchmarks, and best practices. And, as the line between mobile and the desktop continues to blur, we will likely see more dollars being spent in the mobile ad space as mobile content continues to be consumed in an app-centric environment. Customer Content Consumption Continues to Evolve As a result of this shift from classic to online media, the role of the consumer has changed from that of a spectator to an active participant putting customers in control. To add complexity, customers are shifting between devices to experience the best digital experience available. Thus, programmatic buying is the new norm as it delivers messages to end-users with relevant impressions one-at-a-time thereby providing the desired brand experience. However, advertisers need far more insights on how ad tech companies go about spending the ad money and delivering value, even as concerns are raised on how digital data is being used to target customers. Convergence will define success for ad campaigns In a nutshell, the online ad spend will only continue to increase, but as competition grows, ad agencies and advertisers will put greater emphasis on hyper-segmentation to micro-target the right audience. In this scenario, the key to successful campaigns will be linked to content personalization. To achieve this, digital channels, technology, and the growing amount of data need to convulse to provide insights to better target ads. Hence, the goal for all agencies is to use technology and industry knowledge to identify the right data to provide the ideal ROI for advertisers. Thus, the fact to be remembered is that data is not an end in itself but a channel to push the right message using the right media channel at the right time to ensure successful campaigns. The major challenge in this regard is for traditional media companies to be prepared to adapt to the changing ways in which data is being consumed by end-users. Not many media companies are ready for this change though they have sufficient inventory value. What is needed is for these organizations to add value to the inventory rather than depend on the site master-head data to generate ad revenue. At the same time, for organizations ready for the change, their legacy infrastructure might become an obstacle. To summarize, in the complex world of ad technology, more clarity is evolving as time goes by. and as the world changes to a new order for accessing content anywhere, anytime, and on any device, the need now is for an always-on marketing strategy rather than those defined by start and end dates. To ensure success in this new order, advertisers need to have the right mix of channels, technology, and data to ensure a far greater success from ad money spend.
Testing Scenario for Mobile Business Intelligence
Overview Mobile devices have evolved over the years affecting online access to the point where a majority of Internet access is being conducted via mobile handhelds today. To maximize this large-scale market penetration, industry segments have made mobile apps an integral part of their marketing strategy. This accelerated activity has resulted in the growth of Mobile Business Intelligence (Mobile BI) which has transformed the business landscape from a ‘wired’ to a wireless world. Mobile BI is far more versatile than other forms of intelligence as it can be woven closely into people’s movements, work, conversations, meetings, discussions and fun time. Leveraging this versatility, Mobile BI is a package that uses existing BI applications to make informed decisions in real time. What does Mobile BI mean? Mobile BI is the ability to access BI-related data such as KPIs, business metrics, and dashboards on mobile devices. Information delivery suitable for Mobile user interface has been made possible by the various applications provided by Mobile BI. Basic Workflow: The below diagram illustrates the flow of a Mobile BI architecture Comparison of Mobile BI Apps with other options: The table below depicts the difference in effort, interactivity and the ability of various channels in comparison with Mobile BI apps Important points to analyse before opting for Mobile BI Strategy:- See if mobility works for you from a business perspective. See if you have the right IT infrastructure to support Mobile business? Do you have the ability to leverage hybrid (Native + Web-browser) applications to cater to different requirements? Can you account for security based-unique considerations required before its implementation? Best practices for designing the interface in Mobile BI Apps: Avoid dashboard burgeoning – A dashboard should be designed in a way that it provides the necessary information for the decision-maker. It should give easy access to information in a standardized format. The dashboard should be business driven and not purely technology driven. “More is not better”- An abundance of KPIs are not better and leads to overcrowding of the dashboard. Refrain from writing in a smaller form factor – The font size should be in a readable size to ensure that the user does not have to strain his eyes to see the font. The placeholders can be used on the small text or common form inputs like login forms or search boxes.The headings should be kept short to ensure that it does not push the content down the page or out of the frame for users. Strategy for developing and testing Mobile BI Apps: Build once, deploy anywhere strategy With a variety of development languages and approaches, it is apparent that there are varied ways to build regardless of the language. The key is to avoid building the code in a way that it has to be modified for each specific environment. To achieve a `build once, deploy anywhere’ status, it is best to exclude environment specific resources in the final application build so that it is interoperable with the environment. The environment specific artefact can be deployed to the container separately from the WAR and at any time. There are some server-side platforms available for running such applications. E.g.: Oracle WebLogic. Account for New mobile scenarios The evolving mobile solutions landscape has led the field of mobile testing to a challenging level and it has become important for QA managers to understand the unique testing needs of the Mobile BI segment. The QA needs to identify the needs/establish requirements of the various users and design scenarios, taking into account, aspects like interoperability, security and reliability of the mobile app. Pros and Cons of Mobile BI: Pros: User can pinch, swipe, and tap to easily interact with and analyse company data. Cloud based mobile solutions increase collaborations. Allows the user to send notifications in various forms like email or text messages. Provides sales and field support representatives with the data they need to answer customer questions on the spot. Cons: Too much reliance on mobile devices and tablets increases the risk for mobile computing. Devices are expensive and replacement costs are steep. Mobile BI apps in general are not very interactive and restrict users from drilling down into data. Example Snapshots of some BI Apps: Widely used products for Mobile BI: Oracle business Intelligence Mobile:- http://www.oracle.com/us/solutions/business-analytics/business-intelligence/mobile/overview/index.html Tableau Mobile Business Intelligence :- http://www.tableausoftware.com/solutions/mobile-business-intelligence Mobile BI QA The aim of testing BI applications for mobile is to achieve credible data and a good design display with a user-friendly interface. It is also important to ensure that the back-end of a mobile BI system is able to handle the processing load to display data in a timely manner. How to Devise a Mobile BI Test Process: Validate the data required and identify data sources Identify and decide the category of Mobile BI App to be implemented, i.e.: i. Mobile Browser Rendered App ii. Customized App iii. Mobile Client App Understand data, unearth related problems early and identify boundary value conditions for test scenarios. Set up the acceptance criteria as per data accuracy / consistency and benchmark the performance time for rendering of reports. Test plan to identify the scope of testing and prepare test data and testing techniques. Tools which can be used for Testing BI Applications on Mobile: BI Application testing encompasses a lot of systems like data mining, statistical analysis and graphically-rich dashboards. To test all these components tools are available: Eggplant: This is a tool by Testplant. It uses image recognition technology to instantaneously test multiple aspects of Business Intelligence systems. It is also flexible to operate across multiple applications which helps for better testing. Sikuli: Sikuli uses image recognition technology to identify and control GUI components. URL: http://www.sikuli.org/ References: http://searchcio.techtarget.com/essentialguide/Strategic-business-intelligence-for-a-mobile-future#guideSection1 http://www.tableausoftware.com/learn/whitepapers/5-best-practices-mobile-business-intelligence http://en.wikipedia.org/wiki/Mobile_business_intelligence http://wiki.scn.sap.com/wiki/display/BOBJ/Architecture+and+Workflow+Diagrams
Flashback and Looking Ahead: What ‘Tweeples’ Said During the Festive Season?
This New Year, Twitter saw two hashtags trending namely #bestmemoriesof2014 and #20ThingsIWantFor2015 worldwide. In order to find out what people were tweeting in the festive season, in terms of their experiences in 2014 and the emotions they were displaying, I captured and analyzed lakhs of tweets for these two separate hashtags. The tweets for #bestmemoriesof2014 and #20ThingsIWantFor2015 were collected during different time windows on 31 December and 1 January IST when these tags were trending in the Global Top Five List. On applying `Text and Sentiment Analytics’ technology to this data I came up with some interesting results. Take a few minutes to walk with me through peoples’ best memories in the past year and what they wish for in 2015. #bestmemoriesof2014 Findings: Some of the most common words that surfaced include One Direction (British pop boy band based in London), friends, love, Putin, crush and concert. Parsing the tweets to identify the sentiment polarity revealed that over 94% of cities were positive overall barring a few cities like Chicago which were slightly negative in overall sentiment. Along with this, I looked at subjectivity, and the results are as follows: #20ThingsIWantFor2015 Some of the most common words echoing peoples’ wishes for 2015 are hug, new phone, selfie, money, good grades and being a good person. Sentiment analysis for the wishes showed a mixed sentiment with 68% cities having overall positive sentiment and 25% having overall negative sentiment. Tweets for this hashtag were more subjective as compared to the former: P.S.- All the visualizations have been created using Tableau.
Compelling Reasons for Visualization in Retail Trading Applications
An adage goes as: `a picture is worth thousand words’. I am slowly finding out that this can apply very well to the world of retail trading applications. In fact, visualization can become a new way of trading in the future. In some areas, such as technical analysis, a methodology for forecasting the direction of prices through the study of past market data, evolved quickly after the OHLC(Open, High, Low, Close) data of a security was plotted in the form of line and candle stick charts. However, even though the capital markets industry is constantly evolving with innovations and methods of trading, visualizations remain understated. Uses of Visualization Visualization can be very useful in analyzing market data, company results, fundamental information and also news. However, the type of visualization should be carefully selected for each trading app widget, i.e. Watchlist, Option Chain, Order book etc., in such a way that the data represented remains meaningful and tradable. I have always been a fan of www.finviz.com as the visualizations provided by them are very relevant and tradable. However, some of the new features like 3-D heat map are undoubtedly visually appealing, but their relevance and tradability remains questionable. Hence, it is crucial to find the right balance between the visualization type per widget and the data to be visualized. Visualizations developed for a retail trader should be focused to simplify the process, instead of having to skim through tons of data, analyzing them, trading them and finally tracking them effortlessly. In other words, all widgets be it the simple widget like a Watchlist or a complex widget like an Option Strategizer; visualizations should be customizable as per the needs of the trader. Recently, we at Tavant, were working on a project for one of India’s leading bank’s retail trading web application, and the results were studied through web analytics. The response visualizations received from the traders was fascinating! Some of the visualizations like the bubble chart that were provided to analyze market scenarios, and news analytics received significantly more views than the traditional market statistic data like top OI gainer, volume gainers, etc. We also found significant tractions for other visualizations like Fin Map, an interesting, but complex variation of a heat map that could help a trader to analyze a company’s results at a glance. To summarize, there were more views for every visualization implemented on a single trading day. Meanwhile the team at Tavant Technologies, Bangalore, is trying to blend heat maps with technical charts to obtain calendar charts to offer technical analysis to even novice traders. Portfolio Heat Maps In concurrence with the above-defined principles, visualization of a portfolio in the form of a heat map was constructed. Heat map that is one of the many ways of visualizing portfolio was attempted. Heat maps are an easier method to track and analyze a portfolio with colors (red, green or gray) and the area of the rectangle used in the heat map summarizes the portfolio performance at a glance. Traders, on right click, were provided with the option to trade. For advanced traders or portfolio managers, heat maps are constructed with an option of tracking the performance by Invested amount, market value or profit & loss. Drilldowns in the heat map can be used to analyze the portfolio based on asset allocation, sector allocation, and capital allocation. Thus, visualizations are a whole new possibility for retail trading applications where the trader can get rid of numbers, percentages and averages, and trade purely based on colors, shapes, and sizes.
Morphē – Adaptation to Evolution in the Consumer Lending World
During the late 2000s, a set of events, characterized by a rise in mortgage delinquencies and foreclosures, and the decline of securities backed by mortgages, led to the mortgage crisis. House sale prices displayed a steady decline, and as interest rates increased, mortgage delinquencies soared and securities backed with mortgages lost most of their value. The mortgage bubble had burst, and the ensuing crisis had long lasting effects on the U.S. and European economies. Many technology companies which had focussed on this industry segment turned their attention elsewhere. However, long before the mortgage bubble burst, Tavant had already developed an intrinsic relationship with this industry and as a result had acquired deep expertise in providing solutions for achieving higher lead conversion rates, lowering processing costs per loan, optimizing key servicing indicators such as default rates and minimizing the cost of securitization. The company has accumulated more than 2500 person-years of application development experience across the entire mortgage lifecycle. This commitment was reflected in Tavant being recognized in the year 2007 by Mortgage Industry Magazine as one of the magazine’s Top 50 Mortgage Technology Providers. Special emphasis was placed on Tavant’s proven ability to provide a high degree of functional value to mortgage lenders. When the bubble burst, our commitment did not waver. We morphed and accepted the fact that change is the only constant in life. Our commitment to this industry meant that this was not the time to move away but was a period of time for investment. We, therefore, continued to grow our team of techno-functional experts, developers, and architects. When business was slow, and other companies were rerouting talent to other domains, we invested in domain training for our people. They were given domain specific training to understand the eco-system and the way the mortgage industry functioned. Now, the industry is showing signs of recovery, and we are there. However, we have morphed. We are not students waiting to learn from industry experts but are knowledge sharers. Our depth of knowledge about this industry has resulted in a marriage of sorts between technology and functionality. We don’t wait for directions from our mortgage industry customers – we lead discussions. Be the glue and not the hammer Some of the key lessons that we have learnt and which have helped us transform from our role as software service providers to techno-functional experts is, we replace pieces instead of the whole ship, thereby allowing the ship to keep moving. We get into shorter engagements till the end of a project. Where the rule of the game was to go after long engagements, we offer short deliverables that result in long term trust in our abilities to help them take strategic decisions. In the place of multiple systems that are consumer facing, with different benefits and features in each system and each LOB and solution works in isolation, we offer customers increased portal offerings and holistic one-stop-shop solutions We offer customers the ability to plug-in and plug-out components, depending on the business requirements. Thus, to summarize, our investment in domain knowledge is our strength now and the precise reason why some of the biggest names in the industry are partnering with us for strategic solutions which are aligned to their business roadmap. FAQs – Tavant Solutions How does Tavant help lenders adapt and evolve in the changing consumer lending landscape?Tavant provides adaptive lending platforms with flexible architectures, rapid deployment capabilities, and continuous innovation programs that enable lenders to quickly respond to market changes and evolving customer expectations. What evolution strategies does Tavant recommend for consumer lending transformation?Tavant recommends phased digital transformation, customer-centric design thinking, agile development methodologies, and ecosystem partnerships that allow gradual but comprehensive evolution in lending operations and customer experiences. How is the consumer lending world evolving?Consumer lending is evolving toward instant decisions, personalized products, embedded finance, alternative credit data, mobile-first experiences, and ecosystem-based services that integrate lending with broader financial and lifestyle needs. What drives adaptation in the lending industry?Key drivers include changing customer expectations, fintech competition, regulatory changes, technological advancement, economic conditions, and the need for operational efficiency in an increasingly digital world. How can traditional lenders successfully evolve?Traditional lenders can evolve through strategic technology adoption, cultural transformation, customer-centric innovation, partnership strategies, and gradual modernization that leverages their existing strengths while embracing digital capabilities.
MBA’s Annual Convention – The Real Estate Industry Has Definitely Picked up Momentum
The MBA Annual 2014 was exciting and an enriching event as always. It is an event that everyone from the industry looks forward to. To summarise the overall experience and map it to my knowledge, I would say that I found the general atmosphere to be upbeat. There were positive vibes among participants as well as speakers about the economy, the housing market and the overall mortgage industry. Their feelings echoed mine as I observed the trend being positive with unemployment reducing to a healthy level. It is now a purchasers’ market! People will have larger incomes that will encourage them to move to larger houses. This translates to surge in business for the consumer lending industry. I also believe that new constructions will rise to meet the growing demand. This is a period of progress and momentum. The economic growth will lead to increase in demand for housing, which will require lenders to step-up & extend credit to a wide-range of responsible borrowers. In 2015, key trends that will impact mortgage companies will be the level of customer satisfaction, compliance related legal issues, mobile accessibility and warranty management. With regards to the current state of Government Housing Finance, I think that the time is right for reforms to be initiated. For example, if consumers are weighed down by student loans, then this will indirectly affect the market for housing mortgage loans. Some good work has already begun. One of FHFA’s key initiatives is revising and clarifying the Representation and Warranty Framework under which lenders and enterprises operate. These representations and warranties provide the necessary assurances that allow Fannie Mae and Freddie Mac to purchase loans in an efficient and responsible manner without checking each loan individually or being at each closing. They also provide the enterprises with remedies to address situations where lenders obligations to meet the enterprises’ purchase guidelines have not been met completely. To summarize, I believe that the mortgage industry will experience continued growth in the coming years. However, this time around, technology will drive the business. Apart from being solution providers, business & technology partners will need to be domain experts who understand the nuances of this complex industry and offer advanced solutions which are end-customer friendly.