Architecture of a Massively Scalable Distributed ETL System
An Extract, Transform and Load (ETL) tool needs to be robust, scalable, high throughput and fault tolerant. Very much like an e-Commerce transaction system. Designing such a system on a distributed computing backbone can be extremely rewarding, given that mid-size to large organizations might be collecting data from multiple sources and bringing it all together into an integrated warehouse—resulting in thousands of batch and real-time jobs running during the course of a day. For example, retailers collect inventory, sales, finance, marketing, clickstream, and competitor data multiple times a day. But aggregating this data by running ETL jobs, only once daily, can slow down decision-support systems and rules engines, which must feed essential decisions (like dynamic prices) back to the system to control demand. For many e-commerce analytics and data-mining solutions, a slow ETL tool might prove to be a huge bottleneck. While commercial and open source tools help implement such workflows, it is often better to consider a homegrown ETL tool based on good design and distributed-computing principles. Learn how to build your homegrown ETL solution and use a task queue to scale the tool horizontally. Download the whitepaper to read more: http://lf1.me/Ncc/
Automated Mortgages: Just a Click Away
Mortgage lending processes are becoming quicker and user friendly. Thanks to web and mobile technologies, loan applications and procurements are entirely online and do not involve any human interaction. Online-only originators like GuaranteedRate.com, QuickenLoans.com, and Sindeo.com have websites and mobile apps capable of doing everything that the sales agents and mortgage brokers traditionally did. The uncluttered user interfaces and intuitive algorithms can guide all applicants smoothly through the entire process. Looking at the times ahead, every mortgage firm is trying to offer its services online. Many firms are struggling with implementation of their digital versions, and yet, they are wondering why customers are not embracing them widely enough. It’s all very simple: if, at all, you choose to go online, do it right. The last thing you want is embarrassment in a new venture. Here’s the right way to go online with mortgage lending: Wondering what the smart online way is? Self-service online models have intelligent features, and they are designed to keep absorbing more intelligence day by day. Agile development of your online services should make it increasingly user friendly, so that no applicant feels like abandoning the process half way. Let’s see the key differentiators of a good mortgage-lending website: Assistance in planning: Built-in calculators should help mortgage customers try various loan plans, durations, and repayment options, and finally select what best suits their situations. Data collection: a) The system should be able to collect all necessary data, and show up only the necessary data for an applicant. What data is needed depends on the type of loan selected. Difficult or confusing fields should be supplemented with tips and suggestions. b) Photographs, scanned images of signatures, identity cards and other relevant records should be submitted online. All that should be archived with relevant meta tags that help in identifying and retrieving them later. It is important to find the latest credit reports of applicants. Web widgets should be connected to credit-rating agencies and they can fetch the report of the applicants even as the applications are being filled out. You should be able to provide instant responses to applicants as to whether they qualify or not. Long verification processes only discourage mortgage customers. Predictive analytics about the probability of defaults, delayed payments, or non-payments should enable you to offer risk-adjusted rates and nullify the potential risks. Normally, the entire process should be completed and the amount disbursed within 15 days. All the actions need foolproof security and privacy around them. Besides that, personal financial management tools are a great value addition, as they can help customers make wise decisions in a convenient way. The right online implementation of your mortgage banking system can save you from high payroll expenses. Cost and time-efficiency is of utmost importance, and going online should give the client a happy experience. To thrive in a complex and constantly evolving business environment, firms need to improve their systems constantly and maintain the habit of innovation. FAQs – Tavant Solutions How does Tavant make automated mortgages accessible with simple click-based processes?Tavant provides intuitive mortgage automation platforms with one-click applications, automated data verification, instant pre-approvals, and streamlined digital workflows. Their user-friendly interfaces enable borrowers to complete mortgage applications with minimal clicks while sophisticated AI handles complex processing in the background. What automation capabilities does Tavant offer for mortgage lending?Tavant offers automated income verification, property valuation, credit analysis, compliance checking, document processing, and decision-making capabilities. Their platform can process up to 90% of mortgage applications automatically, requiring human intervention only for exceptional cases or complex scenarios. How automated can mortgage processing become?Mortgage processing can be highly automated, with modern systems handling application intake, document verification, credit analysis, property valuation, compliance checking, and initial underwriting decisions. However, complex cases, regulatory requirements, and quality control still require human oversight. What is one-click mortgage approval?One-click mortgage approval refers to streamlined digital processes where borrowers can receive instant pre-approval or preliminary decisions with minimal input, leveraging automated data verification and AI-powered risk assessment to provide immediate feedback on loan eligibility and terms. Are automated mortgages safe and accurate?Automated mortgages use advanced AI, machine learning, and data verification systems that often provide more consistent and accurate decisions than manual processes. They include robust fraud detection, compliance checking, and audit trails while maintaining human oversight for quality assurance and complex cases.
7 Reasons to Invest Wisely in Agile Predictive Analytics Tools
“Here, you see, it takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!” Thus says the Red Queen in Alice in Wonderland. This is very true of the modern business world. We live in times when business advantages are short-lived. Analyzing historical data to plan tomorrows is a kind of sluggish way of doing business. That is why predictive analytics is important. Image credit: commons.wikimedia.org Predictive analytics helps to: Earn low-risk customers Know about developments impeding repayments Reduce service cost and increase profit Provide more individually customized services Run better-targeted marketing campaigns Identify risk events affecting borrowers Improve the maturity of your very analytics These will lead your business to become more agile, competent, and profitable. You acquire some customers. Some of them end up unable to pay back, some turn out frauds, some repay only because of good market conditions, and the rest repay as per the agreement. You analyze this data and assess your overall risk profile, and based on it, you make your future decisions. Now, what about the damage already suffered? What if your analyses and risk profiling were more accurate before acquiring customers and through their repayment period? That is what predictive analytics is all about Predictive analytics software has become an inevitable tool for enterprise risk management in many industries, including banking, insurance, mortgage, healthcare, medicine, travel, and retail. Remarkably, in the parlance of analytics, risk has become almost synonymous with credit risk. The key function performed by a risk-analysis product is transforming uncertainties about the future into probabilities that can be used in business decision-making. Various techniques are used for predictive analytics. Software products rely on multiple techniques, but also on third-party data about customers so that lenders are able to identify risk levels around repayment. Credit scoring and rules-based decision making are important for risk management in financial service organizations. They need actionable and predictive rules that can bring about continuous business growth. By studying the borrowing, spending, and repayment behavior patterns of applicants (individuals and institutions), they can create scorecards. By forecasting the amount to be recovered, schedules for recovery, cost of collection, and methods of recovery, they can strategize the lending and the inventory. Thus, predictive analytics makes businesses agile and competitive. FAQs – Tavant Solutions How does Tavant provide agile predictive analytics tools for lending? Tavant is advancing embedded lending solutions, API-first architectures, real-time decision engines, and predictive analytics for market trends. They’re building platforms that enable instant lending integration across various digital channels and ecosystems. How does Tavant prepare lenders for future fintech disruption?Agile predictive analytics tools are flexible, cloud-based platforms that enable rapid model development, testing, and deployment. They support iterative development processes, real-time data integration, and quick adaptation to changing business requirements without lengthy implementation cycles. Why should financial institutions invest in predictive analytics?Financial institutions should invest in predictive analytics to improve risk management, enhance customer targeting, optimize pricing strategies, reduce operational costs, increase competitive advantage, and comply with regulatory requirements through better data-driven decision making. How do predictive analytics tools improve lending decisions?Predictive analytics tools improve lending decisions by analyzing historical data patterns, identifying risk factors, predicting loan performance, optimizing approval criteria, and providing real-time insights that enable more accurate and consistent lending decisions across all loan types.
Predictive Modeling for Advanced Audience Targeting
Media advertising in this decade is superfast and all about accurate customer engagement. Smartphones, iPads, and the internet bring unprecedented access to information, and publishers are facilitating carefully customized content that caters to old-fashioned as well as new-age audiences. Content is developed with focus on customer needs and brand loyalty. Tailored product or service information is vital. Predictive analytics help publishers understand customers better across all services and brands. With predictive modeling, audience data is sorted for in-depth and actionable insights. That provides recommendations on how to target audiences and engage them whenever required. The objective is always more revenue and greater customer loyalty. Challenges that brands are facing without predictive modelling: Lack of incentive in sharing information across channels and brands Generic product-based information which doesn’t benefit like audience-centric information can Expensive rates for integrated internal database of multi-channel users These cause inaccurate marketing and lead to failure in audience engagement. With cloud-based predictive modeling, brands can achieve what they need. They can target their audiences better and achieve higher ROIs. Here are the reasons: -Analytics reveals customer preferences to develop marketing engagements with exclusive data sorting. This results in tailored inbound and outbound interactions with the most relevant contextual data. -Data-driven mechanics can analyze those elements that drive customer loyalty and customer spend at the micro level. Publishers can invest in customers having the highest potential towards lifetime value. -Analytics can optimize decisions about customer service to improve measures of customer satisfaction and retention. Using historical data, the technology can sort information to identify those elements that churns and retains customers. It further provides the insight that can help in offering proactive service or offer required for customers moving out. – Predictive modelling comes with features like integrating survey information. A similar approach is used to deliver customer experience across all channels. This not only helps in capturing customer responses to enhance the models continuously, but also towards relevant and consistent audience journey. Audience engagement in the digital market has to be relevant, consistent, and personalized. Real-time data is no doubt expensive, but hardly useful without predictive modeling software. Developers can provide technology as well as real-time data cost-effectively, but data alone makes very little sense in terms of cost-to-benefit in the market.
Analytics and Mobility are Tugging Brands into Digital Advertising
The exponential growth in the consumer base for smartphones and tablets is standing proof of the rapid migration of consumers from the world of broadcast, telecast and print media to the digital world. It demonstrates the increasing convergence of the virtual and physical worlds. Juniper’s Digital Retail Marketing Report “Loyalty, Promotions, Coupons & Advertising 2015-2019” has cited the reason for this migration to be the result of timely, targeted, personalized campaigns that enhance customer engagement. Cashing in on this trend, marketers are utilizing the advancement in analytics technology to create ROI enriched marketing campaigns. A forecast by eMarketer predicts that annual mobile advertising spend is expected to grow by more than two-fold by 2018 amounting to nearly $158.55 billion. This indicates fatter marketing budgets which is likely to grow bigger with time, as according to estimates, the global market share of total digital spends is expected to reach 30% in 2015 (Source: Magna Global). The following points highlight the impact and opportunity available for marketers today, owing to mobility and analytics: Maximized Personalization: Provides marketers with the ability to reach users with mobile internet, at the right place and at the right time. By using analytics, marketers can leverage consumer behavior, i.e. Leverage past activity (declared behavior) and interests (undeclared behavior for inferred conclusions by using predictive analytics. Emotional Connect: Content sharing has become easy and ubiquitous with social platforms blurring offline and online interactions. This essentially means brands can facilitate an emotional connection with the end consumer by curating content as per the needs of their target audience. Apart from this, it provides marketers with insights into customer reactions to their products or services providing them with valuable insight into customer preferences. Precise targeting: Abundant user data is available in real-time. This provides marketers with the opportunity to identify potential buyers and offer them impactful tailored content across different communication channels. For marketers to be successful, it will be important to plan seamless digital initiatives with campaigns that capitalize the entire lifecycle across screens and platforms rather than follow a silo model or a fragmented approach.
Right-Time Analytics in Mortgage Lending
The residential lending market has fallen from its peak and has settled at a more realistic area where it is most likely to stabilize in the $1 to $2 trillion mark for the rest of the decade. In the wake of the 2007 crisis, student loans have increased. The Millennials have piled up substantial debts and prefer to rent than buy. House prices have increased in major cities making it even more difficult to buy. Meanwhile, the cost to originate a loan is on the rise. And these are only some of the countless micro and macroeconomic trends currently impacting the mortgage lending sector. Alongside, technology innovations for the mortgage sector have made path-breaking strides to the extent where legacy applications are being replaced by faster, customer-friendly applications. While many companies are focusing on replacing or embracing their existing legacy systems, some of them are also diving deeper to get the best out of technology. These companies are using analytics to provide strategy, growth and revenue-related statistics. However, more often than not, many of these companies invest in analytics mostly to deliver reports to understand past behaviour and use that knowledge to improve processes and bridge existing gaps. The reports are usually sent to decision-makers on a periodic basis or delivered on demand. In most cases, the data is maybe a day old and is refreshed on a nightly basis. These reports serve the limited needs of the company to support its current operations. But to improve efficiency and reduce operational costs, it is essential to provide data at the time when it’s needed the most, not before and not after, this is called ‘Right-Time Business Analytics’. What is Right-Time Analytics? Information on important events which impact the business, have to reach decision makers as fast as possible. For example, if the employee attendance system detected an unusual sign-off for a loan officer who had to submit disclosures to customers on a particular day and if those disclosures have not been reassigned, then alerts need to be fired immediately to the second-in-command or the reporting manager, citing the number of violations that are about to happen. Such alerts, unfortunately, cannot be fired using traditional business intelligence methods where data is loaded into a data warehouse on a nightly basis. For each event like this, the gains may not be as visible to the human eye as it would be when the total number of events is calculated. Though organizations spend a lot of time measuring the average cost per loan, pipeline velocity and cycle time, very few lenders measure and assess the number of hours spent on closing a loan. This is where Right-Time Analysis comes in! Lenders who adapt to right-time business analytics will see natural improvements to operations beyond what is planned strategically. Another important real-time metric that would create a sense of urgency amongst the workforce is a bullet chart that clearly shows the real-time performance of the loans they are working on as compared to the set target and the company average. These are great tools that a company should consider implementing to improve turnaround times in addition to the other regular operational improvements. Social media is another area which benefits from analytics. Analysing user behaviour on websites is critical to detect user grievances and react to the same towards controlling the damage before it becomes viral. Implementing right time analytics along with effective activity monitoring helps identify several areas where operational improvements can be implemented, thus reducing the cost of originating a loan. In a stagnating market, mortgage lenders who recognize the value of Right-Time Analysis will stand to benefit in the short and long term. FAQs – Tavant Solutions How does Tavant implement right-time analytics in mortgage lending?Tavant provides real-time analytics delivering actionable insights at critical mortgage decision points, analyzing market conditions, borrower behavior, and risk factors to optimize pricing and approvals. What specific analytics capabilities does Tavant offer for mortgage lenders?They offer predictive analytics, real-time risk assessment, automated property valuation models, customer behavior analysis, and portfolio performance monitoring to improve decision-making. What are right-time analytics in mortgage lending?Delivery of relevant data and predictive insights exactly when lending decisions need to be made, including real-time market, borrower risk, and property valuation information. How do analytics improve mortgage approval rates?By providing comprehensive borrower profiles, alternative credit scoring, and risk assessment tools to identify qualified borrowers and optimize loan terms. What data sources are used in mortgage lending analytics?Credit bureau data, bank statements, employment records, property databases, market trends, social media insights, utility payment histories, and rental records.
Brace yourself for #MarTech
The AdTech industry has witnessed significant growth in recent years, but there is another industry growing rapidly and which could likely overtake AdTech in the future. MarTech, refers to the innovation in marketing technology outside the context of advertising, focused on experience and data-driven marketing. AdTech, on the other hand, refers to advertising technology focused on advertising and serving technologies such as ad-server and real-time bidding solutions. In the recent past, more and more organizations are adopting MarTech to innovate marketing pipelines, infrastructure, and workflows, to achieve higher operational efficiency, greater insights and better planning for marketing budgets. The image below, published by Scott Brinker, Editor, ChiefMartech, gives an idea about the scale and pace at which the world of Martech, is growing. Scott mentions that the number of companies adopting MarTech has doubled over the last one year with the overall number crossing 2000. Expansion of the MarTech landscape indicates the evolution of marketing, the increasing role of technology in marketing and the heterogeneous nature of MarTech field. Scott draws attention to key areas where innovation and success are clearly visible: Internet – Scott attributes the success of the digital world to the penetration and affordability of the Internet in people’s daily lives. Infrastructure – The transformation of data storage and presentation capabilities has enabled gathering and processing of data for greater understanding of consumer behavior. Scott says that most of this can be attributed to big data, cloud computing, and mobile/web app development Marketing Backbone Platforms – Platforms such as CRM, e-commerce engines, etc. have brought businesses closer to end customers even as they address pain points. Marketing Middleware – Data Management Platforms (DMPs), CDPs, and tag management software have enriched data with metrics that were not available earlier. These metrics have helped to improve operational efficiency and overall productivity for companies. Marketing experiences – Scott calls them the `front-office’ of marketing. These technologies revolve around customer lifecycle such as social media, email, and A/B testing Marketing Operations – Mostly associated with Business Intelligence, Analytics, Visualization and Data Science, marketing operations helps to interpret and find solutions To summarize, even though a lot of action is happening around MarTech, the fact remains that there is huge potential for growth. We will likely witness a shift of focus away from AdTech towards MarTech.
Gaining Intelligence from Gaming – Game Analytics Platform
A successful decades-old, global, video game developer, publisher and hardware company generates several gigabytes of telemetry data on a daily basis. This company has several successful game franchises that attract millions of players on iOS and Android devices every day, and all these clicks and interactions have resulted in incredibly valuable data. If this data can be aggregated and analyzed, it will hold the key to player engagement and retention. An example of the kind of data includes the event streams that indicate when players are playing, duration, levels reached and the money they spend on buying virtual goods such as new levels and avatars. Social networks complement this information with details about players’ real-life preferences. To analyze this data and use it for intelligence an Analytics platform had to be created. Given below are the steps used for building this solution: Solution Architecture The solution was built on Amazon’s cloud platform using Amazon Web Services (AWS). Software Development Kits (SDKs) for the different game technology platforms, including Android and iOS, were used to enable the games to push events data to the data collection server with minimal programming. Data Collection For the purpose of data collection, server using Node.js, which collects high velocity data from players’ mobile devices and writes it in real time to folders on an S3 (Amazon Simple Storage Service) bucket was used. This provides an event-driven architecture and a non-blocking I/O API that optimizes the throughput and scalability of data. An Amazon EMR cluster to process the collected event streams from S3 multiple times every day was adopted. For each batch, a cluster on demand, based on the data volume, wrote the results back to S3, and then shut down the cluster to save on costs. MapReduce jobs validated and cleaned the event data and wrote the results back to S3. Hive jobs then further processed these files to generate facts, dimensions and aggregated facts for later analysis. Data Persistence and Visualization To support rapid query and analysis, the Hive output was loaded into a data mart built on MySQL (and later experimented with Amazon RedShift as well) and Tableau to create dashboards and interactive charts were used. As a result of this solution, the game company gained valuable insights, including: The conversion rates of players from free to paying customers based on geography, game title, and other dimensions The skew in the distribution of paying customers (a small number of players accounted for a large part of the total spending) An understanding of each player’s playtime across multiple games (surfacing opportunities for cross-promotion within each game) Detection of fraud through comparisons of the game’s telemetry data about purchases with the app store data for in-app purchases (it turned out that hackers had exploited a vulnerability in the game design that was quickly corrected)
Data Cloning Through Pentaho Data Integration Clone Step
Splitting rows based on a column value. The input data comprises of ticket booking records defining number of seats booked at event, section and row level. It also contains the starting and last seat number. Objective It was required to split ticket blocks within event_name + section_name + row_name as follows: convert the record where num_seats > 1 into as many records as num_seats assign values as follow in split records num_seats = 1 for each record seat_num = individual seat within the block original seat_num + i where “i” is counter from 0 to num_seats – 1 last_seat = new value of the seat_num as above population logic of rest of the column remains unchanged Sample Data Input: Expected Output: So based on the above screenshots, we need to split the incoming input rows based on the num_seats field .So for first input row where num_seats=4 we need to generate 4 records as per the rules defined above. Solution: Pentaho provides a clone row step that can clone objects or rows in the same way as the main row based on a column value. Refer to the below screenshot for the solution: Table input: This step will load the input data. Clone row: This step will create the clone objects or rows similar to the main row Nr clone in field: will specify column value to be used for cloning Add clone flag to output: will put the flag=N for the original row and Flag=Y for clone rows Clone num field (seat_index_rownum): will add the index value (0,1,2,..). Filter rows: Remove the original (non-cloned) row (where clone?=N). Calculator and Select Values: Calculate the seat number and replace the original fields (num_seats, seat_num, last_seat, etc.) with the new values. Table Output: Loading the data into the target table.
Compelling Reasons for Visualization in Retail Trading Applications
An adage goes as: `a picture is worth thousand words’. I am slowly finding out that this can apply very well to the world of retail trading applications. In fact, visualization can become a new way of trading in the future. In some areas, such as technical analysis, a methodology for forecasting the direction of prices through the study of past market data, evolved quickly after the OHLC(Open, High, Low, Close) data of a security was plotted in the form of line and candle stick charts. However, even though the capital markets industry is constantly evolving with innovations and methods of trading, visualizations remain understated. Uses of Visualization Visualization can be very useful in analyzing market data, company results, fundamental information and also news. However, the type of visualization should be carefully selected for each trading app widget, i.e. Watchlist, Option Chain, Order book etc., in such a way that the data represented remains meaningful and tradable. I have always been a fan of www.finviz.com as the visualizations provided by them are very relevant and tradable. However, some of the new features like 3-D heat map are undoubtedly visually appealing, but their relevance and tradability remains questionable. Hence, it is crucial to find the right balance between the visualization type per widget and the data to be visualized. Visualizations developed for a retail trader should be focused to simplify the process, instead of having to skim through tons of data, analyzing them, trading them and finally tracking them effortlessly. In other words, all widgets be it the simple widget like a Watchlist or a complex widget like an Option Strategizer; visualizations should be customizable as per the needs of the trader. Recently, we at Tavant, were working on a project for one of India’s leading bank’s retail trading web application, and the results were studied through web analytics. The response visualizations received from the traders was fascinating! Some of the visualizations like the bubble chart that were provided to analyze market scenarios, and news analytics received significantly more views than the traditional market statistic data like top OI gainer, volume gainers, etc. We also found significant tractions for other visualizations like Fin Map, an interesting, but complex variation of a heat map that could help a trader to analyze a company’s results at a glance. To summarize, there were more views for every visualization implemented on a single trading day. Meanwhile the team at Tavant Technologies, Bangalore, is trying to blend heat maps with technical charts to obtain calendar charts to offer technical analysis to even novice traders. Portfolio Heat Maps In concurrence with the above-defined principles, visualization of a portfolio in the form of a heat map was constructed. Heat map that is one of the many ways of visualizing portfolio was attempted. Heat maps are an easier method to track and analyze a portfolio with colors (red, green or gray) and the area of the rectangle used in the heat map summarizes the portfolio performance at a glance. Traders, on right click, were provided with the option to trade. For advanced traders or portfolio managers, heat maps are constructed with an option of tracking the performance by Invested amount, market value or profit & loss. Drilldowns in the heat map can be used to analyze the portfolio based on asset allocation, sector allocation, and capital allocation. Thus, visualizations are a whole new possibility for retail trading applications where the trader can get rid of numbers, percentages and averages, and trade purely based on colors, shapes, and sizes.