Product metrics. Quality Metrics and Attributes

Children learn to form word schemes starting in the first grade. However, many children find it difficult to separate form from content; they get confused with symbols and forget the definitions of concepts. Children learn to form word schemes starting in the first grade.

However, many children find it difficult to separate form from content; they get confused with symbols and forget the definitions of concepts. Definition of a word or phrase and its meaning. Scheme. Scheme, -s, feminine gender. 1. A set of interconnected parts of a device, instrument, assembly, as well as a drawing explaining the principles of operation of such a device. General s. node operation. Topic: “Words-objects, words-actions, words-signs.” Objectives: 1. To consolidate the concepts of words-objects, words-actions and words-attributes.

2. Strengthen the ability to correctly ask questions about these words. 3. Develop attention, memory, thinking. 4. Develop skills of organized behavior. Equipment: word-attribute diagram, object pictures, dummies of fruits and vegetables. Progress of the lesson: I. Organizational moment The word-attribute is indicated by the following diagram.

Instructions 2: Now take a lemon. Example: the query “word*” will find words where there are letters: “s”, “l”, “o”, “v”, “o” and one letter is unknown, and all letters can be mixed. If you know the exact position of the letters, the template word search service will suit you. Search in Russian, English and Ukrainian languages. Instant search even in a millionth dictionary ~ seconds.

Huge database of words. More Russian, Ukrainian and English words. On what basis did you guess that words with such a structure are adjectives? Please help me..I will be grateful.

Nyaya (__r.), (___.ch) ___new (___.r), (___ch) ____isty (___.r), (___.ch) _____skaya (___.r), (___.ch) Write the gender and number of adjectives. Under each diagram, write down a word that has the following composition. Ask for more explanation.

Track. Cards in the Russian language "Words-names of objects, signs, actions" card index in the Russian language on the topic. Published - - Murashkina Svetlana Viktorovna. Cards for work in Russian language lessons in grades 1 and 2. Find sign words in the poem and underline them with a wavy line. How the sun walked. In a gold shirt.

A special phonetic feature of words of Turkic origin is vowel harmony - the natural use in one word of vowels of only one row: back a, u or front e, i: ataman, caravan, lasso, chest, heel, mosque, beads. Some grammatical forms of verbs, nouns, and adjectives are characterized by special rules for the pronunciation of sounds in suffixes and endings.

My name is Sergey, I am a Product Analyst at Wikr Group. Before that, he worked in analytics at Philip Morris and Genesis, in the latter he rose to the rank of product analyst. My responsibilities included analyzing all metrics and determining the growth vector. Then he worked at Luxoft for about six months as a business analyst, developing algorithms and microservices. But outsourcing was not as interesting to me as product development, and therefore I soon accepted an offer from Wikr Group. Currently I am developing a mobile application that will solve people’s health problems.

Based on my experience, I want to talk about the position of a product analyst: who it is, what such a specialist does and how to develop in this direction.

Working with data

A product analyst is first and foremost an analyst, which means he must be able to work with data. The more data, the higher the likelihood of making the right decision. To do this, you need to study metrics, build funnels, and monitor what results the slightest changes lead to.

This approach - making decisions depending on the data received - is called Data Driven Development. For example, there is a question: which button will users be more willing to click on - red or green? With the DDD approach, the answer is found using A/B testing: they launch a test, dividing the audience 50 to 50. Half are shown a red button, half - a green one. If the result turns out that users click on the second one more often, then we decide to remove the red one and leave only the green one. Thus, we take only justifiable actions and constantly improve the product.

It is worth considering that at this stage you cannot simply trust the larger value. Product analysts use a measure of statistical significance. For example, what will be better in the case of an uneven distribution of traffic: a sample of 10 users, of which 2 took the target action, or a sample of 100 users, of which 10 made the target? If we rely only on relative indicators, it will be clear that the first sample of 2/10 is better. But in fact, at the current stage it is impossible to draw conclusions, since the conversion to the target action of 20% may be an accident.

With large samples, we obtain results with a high level of statistical significance. The norm for decision making is considered to be a level of this indicator > 95%

One of the most important indicators for a product line is user return. Of course, you can and should scale your traffic and attract new customers. But efforts to retain existing customers and encourage them to use your service again often bring greater ROI. This is where big data analysis also comes to the rescue.

Once, as the head of the user return department, I was faced with the task of reducing the spam rate of the mailing list. At the entrance, I had a deliverability rate of 80%, a spam rate of 12-15%. This spam rate is considered too high. As a result of the work, it was possible to reduce it to 5%, then stabilize it at the level of 7% - almost 2 times. To achieve such indicators, I understood each metric, launched and tested funnels, and turned off certain emails.

An interesting and working approach is to transfer users within your own ecosystem of products. In this case, the user is lured from one company product to another - for example, from a content project about political news to a resource dedicated to interesting facts about travel. Such manipulations are carried out at the moment when the user loses interest in the first project, and the probability of returning him to the current resource is close to zero. To do this, they use the RFM model of user return: based on data on returns and spending of money, as well as the frequency of user spending, one or another interaction logic is applied.

If a user has not returned to a product for a long time, some companies are ready to “invest” in it: incur losses now in order to achieve customer loyalty in the future, moving it to a higher segment. The higher the segment, the higher the investment in the user for his return - from a simple discount to a material gift.

Action strategy based on RFM analysis (stands for recency, frequency, monetary)

Main goals

The responsibilities of a product analyst may vary depending on what stage the product is in. For example, if you just came to the project, the first task may be to build reporting - first of all, for yourself. It is important to understand where the data comes from, how it moves, and how to perceive this or that metric. Then you can build action plans and think about how to improve performance.

Study. At my current job, I was entrusted with creating a product from scratch. My first task was to research the market, analyze competitors, and decide which market to enter first. For example, the cost of a user in the States is insanely expensive, and therefore, to launch the first version of the product, it is better to pay attention to the CIS countries, Latin America, and India. Traffic in these countries is cheaper, and they are more suitable to test the beta version, see how it goes, where users will click. The first version is based on assumptions and analysis of competitors: we assume which functionality will be needed and which will not. But after testing, you can move on from guesswork to specific indicators.

Strategy. After market research, it’s time to build a product development strategy for a year and a half, describe the formation of an ecosystem, and estimate the time costs for each stage of development. At this stage, it is very important to properly design the data architecture. During the development of a strategy, there are a lot of iterations and communications with everyone - developers, designers, recruiters, marketers. Sometimes we also bring in third-party consultants to determine what can be used and what cannot according to the policies of the partners through whom we will promote the product.

Why is it important to build a strategy for such a long period of time, and why can’t you get by with the next few months? First of all, this is necessary for choosing development technologies. Depending on the functionality that is supposed to be integrated at future stages, a platform and technology stack are selected. For example, now it seems like a good option for you to use React Native to create a mobile application, but if this application takes up too many graphic resources, then in a year and a half you will simply hit the ceiling and find yourself hostage to the situation. It's the same with databases. MySQL is open source and convenient solution. But if you plan to store data on the actions of hundreds of thousands of users and build neural networks, then in 3-4 months you will have to redo everything from scratch. Therefore, it is better to think through everything in advance and immediately foresee the scale to which your audience will grow.

Specification and data schema. At the initial stage of the product life cycle, the product analyst develops a document that contains a basic description of the product ecosystem: what will be available on the website or in the mobile application.

The specification also includes a description of the data schema: the product analyst is the initiator in its construction. He can independently design the data schema or act as a customer for the analytics department. It is necessary to think about what metrics will be needed to develop the product, what exactly needs to be tracked, what user actions to study. For example, you can schedule logging of transitions to a certain page in order to understand whether it is worth leaving it in the future or whether it would be better to replace it with something else. In this case, the analyst is the eyes of the product.

Working with the team. Having written the primary specification, the product analyst passes it on to the business/systems analyst, who transforms the document into more specific specifications for development. The entire life cycle of a product is tied to the cooperation of specialists in different fields. After the product strategy and specification have been adopted, designers, layout designers, developers, and testers need to be involved in the work.

No matter how detailed the specification is, team members will ask questions, and therefore you need to be prepared for constant communications.

Product improvement. After the first release comes the turn of constant work on how to improve the product. A/B testing, reporting, analytics - and it becomes clear what exactly needs to be changed or improved. You won't get a good product the first time :) First they release a service with basic functionality, then they refine the core features and add new ones. For example, what is primary for a dating site? Questionnaires, chat. And only then comes the turn of additional services - virtual/real gift systems, etc. The product analyst answers the question of what to implement and in what order.

Decisions to improve the product are made based on the output of the new functionality and the time spent on its implementation. You can use the ROI indicator - the ratio of the expected revenue increase to the development time spent. The service with the highest score is implemented first.

At the stage of integrating new functionality, it is important to correctly estimate the time. In addition to implementing the functionality itself, you need to spend a certain number of hours creating an infrastructure for informing the user (letters, banners on the website), prioritizing and timing the launch of parts of the ecosystem. For example, it would be incorrect to inform the user about a new functionality in an email a month after its launch.

The main thing is to see in advance the path where you are going: what will be integrated at each development phase. At the same time, constant tests adjust the original plan: maybe some user behavior in phase No. 2 will simply cancel the launch of phase No. 7.

Tools

I work with Tableau - it is a data analysis and visualization tool. It is convenient for building statistical reporting - for example, to see what kind of return we have today, how marketing pays off. The service allows you to build user conversion funnels at certain stages: landing → registration → subscription to the trial version → payment → re-payment, etc. All funnel indicators are relative.

For deeper analysis I use Python. It is capable of processing very large data - billions of rows and above. Excel cannot cope with such volume. Also, using Python, you can build various models, carry out clustering according to certain parameters, and simulate user behavior to test the speed of services. In general, knowledge of programming is not necessary for a product analyst, it’s just my hobby.

For research, I sometimes use SimilarWeb - this service provides a small cross-section of the audience, which is extrapolated to the entire sample. The tool allows you to see where your competitors are currently active, where they are currently trying to enter, and how much of an audience they have. This information helps you set your own vector of development.

Portrait of a good analyst

Systematicity. I used to like to rush into problems, but then I realized that every idea takes time to finally mature. There is no need to run with a torch: “Here, I came up with an idea.” Think through all the pros and cons of this idea, formulate a systematic work. A good practice is to plan tasks for a week in advance, write them down in a notebook and gradually complete them, without rushing from one to another.

Attention to detail. You must be familiar with the data and understand what indicators come from where. This will allow you to avoid mistakes or identify them as quickly as possible. For example, I once managed to find a cheater on our product based on traffic sources: I discovered that 90% of registrations came from the same IP address. It turned out that our partners were deceiving us. Attention to detail allowed the system to be identified on the second day, saving the company tens of thousands of dollars in the long run.

Strategic thinking. You need to be able to look at the entire project “from above.” Personally, analytics and computer science helped me achieve this.

Every split and decision made must first of all bring value and benefit to the business. At the same time, you can’t get hung up on one narrow functionality. You need to understand that if you are not implementing a complex and smart system now, then only so that you can launch a simpler version of the product as soon as possible and collect data on user reactions, adjusting your future course. This is much more rational than creating a complex system for a year without a “baptism of fire.”

Good communication skills. Every idea needs to be “sold” both to the user and to your team.

Restraint. Sometimes you have to give in, because your point of view will not always be the most optimal.

Openness. Never assume that you know the users. They are completely different!

career paths

As a rule, all product companies have product analysts. In outsourcing, this position is rare: the product is most often presented on the customer’s side. On the executor's side, this will be a business analyst working with requirements.

Product analysts grow from analysts, marketers, traffic purchasing specialists, and managers. It is desirable to have basic knowledge of mathematics and statistics, and be able to work with analytical tools. Knowledge of SQL will be a plus.

The profession of a product analyst is open to anyone who is attracted by the opportunity to create a product, sculpt from clay what you want, and most importantly, what the market requires at this moment.

Among the interesting resources I can recommend:

• PythonProgramming.net is an excellent free resource that covers current topics in machine learning, statistics, and basic Python.
• Badoo dating channel- here you can find interesting lectures on product and email analytics, in particular by Andrey Sas.
• Faculty of Computer Science HSE- Yandex School of Computer Science.

From offline: Genesis IT school is a good place to start, where you can learn about all facets of big business. I give talks there on user return and mailing.

Currently, a system of metrics has not yet been fully formed in software engineering. There are different approaches to determining their set and measurement methods. A measurement system includes metrics and measurement models that are used to quantify software quality.

When determining software requirements, the corresponding external characteristics and their attributes (subcharacteristics) are specified, which determine different aspects of product management in a given environment. For a set of software quality characteristics given in the requirements, the corresponding metrics, models for their assessment and ranges of their values ​​are defined to measure individual quality attributes.

According to the 1BO 14598 standard, metrics are determined by a model for measuring software attributes at all stages of the life cycle (intermediate, internal metrics), and especially at the stage of testing or functioning (external metrics) of the product. Let us dwell on the classification of existing software metrics, the rules for conducting metric analysis and the processes of their measurement.

There are three types of metrics: software product metrics, which are used to measure its characteristics or properties; process metrics that are used to measure the properties of the life cycle process of product creation; usage metrics.

Software product metrics. These metrics use external metrics, which denote features of the product that are visible to the user, and internal metrics, which denote features that are visible only to the development team.

External metrics software product:

• reliability metrics, which are used to determine the number of defects;
• functionality metrics, which help determine the presence and correct implementation of functions in the product;
• maintenance metrics that measure product resources (speed, memory, environment);
• product usability metrics that help determine the degree of accessibility for study and use;
• cost metrics that determine the cost of the created product.

Internal metrics software product:

• size metrics needed to measure a product using its internal characteristics;
• complexity metrics needed to determine product complexity;
• style metrics that serve to determine approaches and technologies for creating individual components of a product and its documents.

There is also a certain general measure - degree of traceability PP, which is determined by the number of traces traced using scenario models of type UML, and an assessment of the number of requirements, scenarios and actors, objects included in the scenario.

Internal metrics measure product performance and are relevant to external metrics.

External and internal metrics are set at the stage of forming software requirements and are the subject of planning and management in the process of achieving the quality of the final software.

Product metrics are often described by a set of models to set various properties, quality model values, or predictions. Measurements are usually carried out after calibration of metrics in the early stages of the project.

• a measure of software size in different units of measurement (number of functions, lines in a program, disk memory size, etc.);
• a measure of time (system functioning, component execution, etc.);
• measure of effort (labor productivity, labor intensity, etc.);
• accounting measure (number of errors, number of failures, system responses, etc.)

A special measure can be reuse level components of a software system, measured as the ratio of the size of the product made from off-the-shelf components to the size of the system as a whole. This measure is also used to determine the cost and quality of software. Examples of such metrics include the following characteristics: total number of objects and number of reused ones; total number of reused and new operations; the number of classes that inherit specific operations; the number of classes on which a given class depends; number of class/operation users, etc.

When estimating the total quantity of certain quantities, they are often used average metrics(average number of operations in a class, class descendants or class operations, etc.). An example of such widely used external metrics are Halstead metrics - these are characteristics of programs identified on the basis of the static structure of a program in a specific programming language, for example, the number of occurrences of the most frequently occurring operands and operators, the length of the program as the sum of the number of occurrences of all operands and operators, etc. Based on these attributes, it is possible to calculate programming time, program level ( structure, quality) and programming language (level of abstraction of the language tools used, degree of orientation to the problem), etc.

Typically, the metrics used are largely subjective and depend on the knowledge of experts making quantitative assessments of the attributes of the software product components.

Process metrics. These metrics can be used such as development time, the number of errors found at the testing stage, etc. But in practice, the following process metrics are usually widely used:

• total development time and separate time for each stage;
• model modification time;
• time for performing work on the process;
• number of errors found during inspection;
• cost of quality control;
• cost of the development process.

Usage metrics. They serve to measure the degree of satisfaction of the user's needs when solving his problems; they help to evaluate not the properties of the program itself, but the results of its operation - its operational quality. Examples include the accuracy and completeness of the implementation of user tasks, the resources spent on effectively solving user problems (labor costs, productivity, etc.).

Standard assessment of quality indicators. In accordance with the considered four-level quality model, software quality assessment begins from the lowest level of the hierarchy, i.e. from the most elementary property of the quality indicator attribute being assessed according to established measures. At the design stage, the values ​​of the evaluation elements are established for each attribute of the quality indicator of the analyzed software included in the requirements.

According to the definition of standard 180/1E89126-2, a software quality metric is “a model for measuring an attribute associated with an indicator of its quality.” When measuring software quality indicators, standard 180/1E89126-2 recommends using the following types of measures:

• measures of size in different units of measurement (number of functions, program size, amount of resources, etc.);
• time measures, periods of real, processor or calendar time (system operation time, component execution time, use time, etc.);
• measures of effort, productive time spent on project implementation (labor productivity of individual project participants, collective labor intensity, etc.);
• measures of intervals between events, such as time between successive failures;
• counting measures, counters for determining the number of detected errors, the structural complexity of the program, the number of incompatible elements, the number of changes (for example, the number of detected failures, etc.).

Quality metrics are used to assess the quality of a program (failure-free operation, feasibility of functions, usability of user interfaces, database, etc.) using data obtained after testing on a variety of tests.

In testing, the most important metric is the time between failures, which, as a reliability attribute, determines the average time between the appearance of security threats and provides a difficult-to-measure estimate of the damage caused by the corresponding threats.

Very often a program is evaluated by the number of lines. When comparing two programs that implement the same application task, preference is given to the shorter one, since it is created by more qualified personnel, it has fewer hidden errors, it is easier to modify and takes less time to debug and modify, although it costs the same as usually more expensive. Thus, program length can be used to compare and evaluate programs based on the skill of the developers, development style, and environment used.

If the software requirements specified the use of several indicators, then each indicator calculated after data collection is multiplied by the corresponding weighting factor, and then all indicators are summed up to obtain a comprehensive assessment of the software quality level. Based on measuring quantitative characteristics and conducting an examination of qualitative indicators using weighting coefficients, the final assessment of product quality is calculated by summing up the results for individual indicators and comparing them with software benchmark indicators (cost, time, resources, etc.).

When assessing an individual indicator, a weighting coefficient-metric, coefficient-indicator, and coefficient-attribute are calculated using evaluation elements. For example, let's take portability as an indicator. This indicator will be calculated based on five known attributes, and each of them will be multiplied by the corresponding coefficient. All attribute metrics are summed up and form a quality indicator. When all attributes are assessed for each of the quality indicators, a total assessment of the individual indicator is made, and then an integral quality assessment is made, taking into account the weighting coefficients of all software indicators.

Ultimately, the result of quality assessment is a criterion for the effectiveness and feasibility of using the design methods, tools and methods for assessing the results of creating a software product at the stages of life cycle.

According to the DSTU 3230-1995 standard, the following methods are used to assess the values ​​of quality indicators: measuring, registration, calculation and expert (as well as combinations of these methods).

Measuring method is based on the use of measuring and special software tools to obtain information about software characteristics, for example, determining the volume, number of lines of code, operators, number of branches in the program, number of entry/exit points, reactivity, etc.

Registration method used when calculating the time, number of failures or failures, the beginning and end of software operation during its execution.

Calculation method is based on statistical data collected during testing, operation and maintenance of the software. Calculation methods are used to evaluate indicators of reliability, accuracy, stability, reactivity, etc.

Expert method carried out by a group of experts - specialists competent in solving a given problem or the software used. Their assessment is based on experience and intuition, and not on the results of calculations and experiments. Such examination is usually carried out by reviewing programs and accompanying documents; For this purpose, controlled signs are established that are correlated with one or more quality indicators and are included in the expert survey cards. The method is used to evaluate such indicators as analyzeability, documentability, structured software, and contributes to a comprehensive and high-quality assessment of the created product.

When assessing the values ​​of quality indicators, depending on the characteristics of the properties they use, the methods of their determination and purpose, a certain measurement scale:

• metric scale (absolute, relative, integral);
• ordinal (rank) scale, which allows you to rank characteristics by comparison with reference values;
• a classification scale characterizing the presence or absence of the property under consideration in the software being evaluated.

Indicators that are calculated using metric

scales are called quantitative, and indicators determined using ordinal and classification scales - high quality.

• 1. The nominal scale reflects the categories of properties of the assessed object without their ordering.
• 2. An ordinal scale is used to order characteristics in ascending or descending order by comparing them with basic values.
• 3. The interval scale specifies the essential properties of an object (for example, a calendar date).
• 4. The relative scale specifies a certain value relative to the selected unit.
• 5. The absolute scale indicates the actual value of the quantity (for example, the number of errors in the program is 10).

Cohort analysis is an effective tool for product and marketing analytics. Even those who know about its existence use it extremely rarely. As part of the “Analytics Course” series of articles, an analyst from ZeptoLab will talk about the effectiveness of cohort analysis Oleg Yakubenkov.

Let's try to compare two cars and find out which one is better:

• the first drove 2,000 km, the second - 12,000 km;
• the first car is used 5 times a week, the second - 4 times;
• The first car traveled on average 10 km in the last month, the second - 20;
• at this particular moment, the first car is traveling at a speed of 100 km/h, and the second car is traveling at a speed of 70 km/h.

Unfortunately, it is impossible to answer the question posed based on the available information. For some reason, as soon as it comes to Internet projects or mobile applications, everyone begins to monitor metrics like DAU, MAU, income, total number of registrations and try to draw conclusions based on them about the product, the impact of changes and the effectiveness of marketing activities.

The metrics listed above are growth metrics. It is useful to follow them for a general understanding of the situation, but in relation to work on the product they are useless, since on their basis it is impossible to make product decisions, just as it is impossible to evaluate the impact of product changes.

A product manager should be primarily interested in its “volume” and “density,” not its “mass.” The “mass” simply states a fact without explaining where it came from or how to influence it. We need to strive to decompose key metrics into their components, decompose them, identifying levers of influence on them - the main task when working on a product.

This activity cannot be done without analytics. Analytics is feedback on actions, eyes in the product world. First, analytics allows us to understand where we are, what kind of product we made, how it is used in the real world, and then it allows us to see how actions and changes made affect the product. In the picture below, I call analytics the stages: Measure, Data, Learn.

One of the most effective product analytics tools is cohort analysis. This is exactly what we will talk about today.

Why growth metrics are meaningless for product analytics

Let's consider the following model situation. There is a product that has the following characteristics:

• the cost of attracting a user is $1;
• Average revenue per user is $2 over the next 4 months;
• 30% of new users continue to use the product after a month (then the share gradually decreases to 15%);
• the promotion team will attract 10 thousand new users in the first month after launch, 15 thousand in the second, 20 thousand in the third, and so on;
• the product manager, who is responsible for the development of the product, makes changes to it every month. The changes are unsuccessful, so after each change, revenue per user drops by $0.1, and the share of users who continue to use the product drops by 2%.

In the company where this product is being developed, it is customary to monitor the monthly audience (MAU or Monthly Active Users) and the profit of each project. Based on these metrics, KPIs are set and the success of the team working on the product is assessed.

Following the selected metrics, after the first 9 months, management was very pleased with the results of the new product, including the success of the product manager. But remember - our product manager spoils the product every month! At the same time, growth metrics are steadily rising.

Below are the same graphs, but for 16 months. In these graphs we finally see the first signs of unsuccessful product changes. But only after 12 months.

The fact is that growth metrics are influenced by two components: product and promotion. When looking at growth metrics, you can't simply separate the two. It is for this reason that growth metrics are completely unsuitable for product analytics.

With properly constructed analytics, we would have seen the unfortunate impact of product updates in the first weeks/months.

The essence of cohort analysis

On any given day, your product's audience is a mixture of those who started using your service today, yesterday, a month ago, and so on. Monitoring this heterogeneous mass and trying to draw conclusions is an extremely thankless task.

The idea of ​​cohort analysis is to divide users into groups based on certain characteristics, and track the behavior of these groups over time.

Typically, groups of users (cohorts) are identified based on the week (month) when users came to the application. Once we identify these user groups, we track them over time and measure key metrics for each individual cohort. By comparing the performance of the March and May user cohorts, you can objectively compare the product versions corresponding to these time periods.

For deeper analytics, the identified cohorts need to be further segmented based on traffic source, platform, country, and other factors that make sense for your specific product.

Most likely, the values ​​of key metrics will differ for different segments, just as different product changes will have different impacts on different user segments.

Key product metrics - LTV and CAC

Two key metrics that ultimately determine the financial success of your product are LTV (Life Time Value) and CAC (Customer Acquisition Cost).

LTV is the money the average user spends on your mobile app over the entire time they use it. CAC is your cost to attract an average user.

Why these two metrics are so important for your product and how they affect your business performance you can read in the material “SaaS Analytics. Criteria for Viability" and in the material "Startup Killer: Cost of Customer Acquisition" or look at Vimeo. For the purposes of this article, the importance of these metrics will be assumed by default, and ways of working with these metrics will be covered in more detail.

LTV is a key metric that reflects the value (benefit) of your product for your users and customers. It is this metric that should be at the forefront when working on a product.

LTV is a great metric, but it has one drawback - it is high-level. To understand how to influence it, it is necessary to decompose it into simpler and product-specific metrics.

Decomposition of LTV into product metrics

Typically, metrics are tied to key points in the user lifecycle of an application. Thus, we create the ability to track the success of user promotion in the application and find bottlenecks that require our attention.

I typically track the user journey through a product in terms of engagement and monetization.

Engagement is described by the following stages in the user lifecycle:

1. activation in the application
2. stuck in the application (or usage activity)
3. long-term retention (how many users continue to use the product a month, two months, and so on after registration)

Monetization is described by the following sequence of stages in the user life cycle:

1. activation in the application
2. I saw a sales screen
3. made 1 purchase
4. made 2 purchases

Below I have provided metrics corresponding to each of the stages of the user life cycle in the product (metrics may differ for different products):

Activation in the application (% of those who completed the tutorial or performed a key target action in the application, for example, registered and added their first friends);

• app stuck(% of users who reached level N or, for example, added N friends: the number N is determined experimentally);
• the user saw a purchase offer(% of users who saw the purchase offer);
• the user made the first purchase(% of those who buy something in the application, average amount of the first purchase);
• the user made a repeat purchase(% of those who made a repeat purchase, average amount of repeat purchase, average number of repeat purchases);
• retention(% of users who use the application a month/two/three/four after registration).

All these metrics ultimately influence LTV. Each product may have its own specifics, but for most, these basic steps/metrics will work.

Product metrics and how they affect LTV

Let's look at the product metrics described above and how they affect LTV using an abstract game as an example.

Activation in the application

In any game, the user is first trained by walking him through a tutorial. Those who have not completed the tutorial will most likely not continue to play, much less pay. This is why it is critical for us to track the percentage of users who successfully complete this stage.

It is also useful to track the percentage of those who were able to complete a number of targeted actions after completing the tutorial (that is, they learned and can now play independently). This metric will reflect how well the learning process is designed.

The user is stuck in the application

The user will most likely not pay if he is not interested in the game. This is why we need to track the share of those who play the application constantly. To this end, we measure the percentage of those who progressed to level N or those who logged into the application more than 5 times within a week of installation.

Typically, the metric for the fact of sticking is determined empirically (examples of such metrics for a number of popular services).

The user saw the purchase offer and made the first purchase

One of our goals is to generate revenue, so we need to encourage the first in-app purchase. But the purchase is made from a specific screen in our application (for example, from the store screen), so we need to track the percentage of users who saw this screen.

If the sale screen is seen by 10% of incoming users, then this automatically limits the share of users who can make the first purchase in our game.

Repeat purchases

A first purchase is good, but financially successful products tend to have a high rate of repeat purchases. Often, the first purchase is a certain amount of trust from the user to the application - if he is satisfied with the result and the benefits received, then, most likely, he will make a repeat purchase. Therefore, another important metric is the share of users making repeat purchases, as well as the average number of repeat purchases.

Retention

In order for users to have a chance to make multiple purchases, they must continue playing our game for a long time, rather than quitting after a day. To track this phenomenon, we will measure retention.

Building product analytics and an example of using cohort analysis

The easiest option to implement for building product analytics would be to create funnels for each of the events described above. In most cases, you will end up with a monetization funnel and a user engagement funnel.

Next, you need to compare your product's performance across cohorts of users based on the week they came to the app. Mixpanel and Localytics tools are ideal for such analytics.

A more complex, but also the most productive approach is the use of cohort analysis for product analytics. Using cohort analysis will deepen your understanding of the product and how your users use it over time.

We will form cohorts of users based on the week they came to the application. For simplicity, the example considers only the following metrics: CAC, LTV, Ratention, % of those who made the first purchase, % of those who made a repeat purchase. Also, for simplicity, the cohorts were not segmented by any additional characteristics.

Below is a table of cohort analysis of the product in question (you can think of it as a game or a travel app).

In the first week, 3,000 users came to the first version of our application. At the end of “week 0”, 25% of them completed the tutorial, but no one had paid yet. By the end of the first week, another 5% completed the tutorial (that is, a total of 30%), while 1.2% made their first purchase. By the end of the second week, 34% of the cohort in question completed the tutorial, and 1.4% made their first purchase.

A week later, we released a new version of the application, where we changed the tutorial. As we can see from the cohort analysis table, it worked! By the end of the fourth week, 47% had completed the tutorial (previously only 34%). Expanding the monetization funnel at the tutorial level also increased the share of those who made a purchase. Unfortunately, our users do not make repeat purchases, which prevents us from reaching operational breakeven for the product, even though the promotion team was able to significantly reduce CAC (albeit by reducing the influx of new users). We spend $0.8 on acquisition, but earn only $0.5 from the average user after 8 weeks.

In the third version of the application, we improved the tutorial and added new purchases to the application, increasing the variety. This allowed us to increase the share of repeat purchases and equalize LTV with CAC.

This is roughly how cohort analysis allows us to understand our product and which improvements work and which don’t.

In custody

The most difficult stage in working on a product occurs when the first metrics values ​​for your product are received and questions arise:

• Are the obtained metric values ​​good or bad?
• Which metric should you work on first in the next version of your application?
• How to prioritize the hypotheses invented to improve the metric?

You can only interest an investor with numbers. What indicators will lead to billions of dollars in investment in your company? The Andreessen Horowitz venture fund, whose portfolio includes BuzzFeed, Facebook, Foursquare, Skype and others, described what metrics its partners pay attention to, and The Secret selected the most important of them. We did not include the most common indicators such as the number of active and registered users, gross profit, traffic sources, cost of customer acquisition or average check, but focused on those metrics that you are unlikely to use.

Inventory Turns / Inventory Turnover

Everyone knows how to calculate conversion: the quantity of goods sold in a certain period must be divided by the quantity of goods at the beginning of the period. The higher this indicator, the more in demand the company’s products are and the more attractive the company is to investors. For marketplace businesses, high conversion means that suppliers will continue to work with it. Investors will also be interested in conversion growth - it is especially important in the early stages and shows the development of the business.

For retailers and manufacturers, high conversion rates indicate that they are able to correctly anticipate demand and meet customer needs. But for these categories of businesses, sometimes another indicator is more important - inventory turnover. The calculation algorithm is as follows.

1) We need to find out the average cost of inventory. For example, at the beginning of the year you had goods worth 100,000 rubles, and by the end - 120,000 rubles. Both figures are added and divided by two - the average cost of inventory is 110,000 rubles.

2) Now we calculate inventory turnover. The cost of goods sold for the year is divided by the average cost of inventory. For example, if you sold goods worth 1.5 million rubles, the turnover ratio is 13. The higher the number, the more efficient the company.

If turnover decreases over time, this is a bad signal for investors - either demand is falling or the company is not developing. There are two ways to increase this ratio and make the company more attractive to investors: sell more intensively with the same amount of inventory, or sell the same, but reduce the amount of inventory needed to complete transactions.

Network Effects / Network effects

A network effect occurs when the value of a product increases as more people use it. It is impossible to calculate it using one formula - for each business it is important to determine its own metrics that will show the effect. This is necessary not only to tell the investor that you have created a community of clients, but also to confirm this with facts.

Andreessen Horowitz cites the example of OpenTable, a service for online reservation of tables in restaurants. Their network effect is that a greater selection of restaurants attracts more customers, and in turn, more customers mean more new restaurant partners. Here are the metrics that helped confirm this effect for investors.

1) Productivity of OpenTable salespeople is increasing, in part due to an increase in the number of requests from restaurants. This is more important than the growth of the restaurant base itself, since it grows without network effects.

2) The number of customers visiting restaurants through OpenTable is growing. This is again more important than the number of registered users.

3) The number of customers coming to OpenTable to make a reservation is growing compared to the number of customers making it through the restaurant's website.

4) Finally, the churn rate of partner restaurants is decreasing.

Each business will have its own set of metrics, but it is important for an investor to see that you are not just observing this network effect, but that you can manage it.

Virality

If network effect is a measure of the value of a customer community, then virality is a measure of the speed with which a product spreads from one user to another. The virality rate shows how many users have attracted other people to use the product. Here's a basic way to calculate this ratio.

1) Count your current users. Let's say there are 1000 of them.

2) Multiply this number by the average number of invitations that one user sends to his friends over a certain period of time. That is, if a user sends about five invitations to friends per month, the total number of invited users is 5000.

3) Determine how many invited users performed the desired action. Be careful with this point - for example, the number of downloads of an application does not mean that new users launched it. It is better to choose those who have completed, for example, the first level of your game. Let's say it turns out to be 15%, or 750 people.

4) The viral coefficient can be calculated by dividing the number of new users by the number of old users. In this case it is equal to 0.75. A result less than one is not considered viral. If you scored higher, you managed to hook users, and your costs for attracting a new client will be lower than those of your competitors.

It is important to remember that virality does not indicate the presence of a network effect - quantity does not always translate into quality.

Net Promoter Score (NPS) / Loyalty Index

This metric will allow you to find out how satisfied the consumer is with your services. It is based on the main question: “What is the likelihood that you will recommend our company/product/service to a friend or colleague?” Ask clients this question and allow them to answer on a scale of 1 to 10 (10 being most likely). Then find out the percentage of “promoters” (those who gave 9 to 10) and “detractors” (those who gave ≤ 6). The consumer loyalty index is equal to the difference between the percentage of “promoters” and “detractors”.

For an investor, a high consumer loyalty index is an indicator that the company thinks not only about the number of users who performed a certain action more than several times a month, but also has good development prospects. Investors can also compare your loyalty index with your competitors.

Cohort Analysis / Cohort analysis

Cohort analysis allows you to identify how users interact with a product over certain periods. For example, you can take for research a group that registered with your service in the first week of January, and track which of them uses the service after a month, three months, six months and beyond. Investors especially value this because most startups don't make a profit and only user behavior can help gauge a company's prospects.

Here are the most important steps for proper cohort analysis.

1) Choose the right set of metrics, not metrics that show off (for example, number of downloads).

2) Choose the right period for analysis - usually a day, a week or a month, because shorter time periods are better for young businesses.

3) Combine the period and metrics - at this time, 100% of your users take a certain main action (buy a product, post a photo, and so on).

4) Select a second period - a week or a month later - and check how many users are still performing this action.

5) Repeat the analysis at other times to understand what influences user behavior.

Investors need to see that the percentage of customers retained (or customers taking a particular action) is stable or growing over different periods. This means the business is on track to build a large, sustainable customer base. Investors will also like it if new cohort analyzes (for example, after the introduction of a new feature) show better results than previous ones. This suggests that over time the company is adding value to the product and learning to better understand its users.

Customer Concentration Risk

Customer concentration is determined by the ratio of revenue from the largest customer to total revenue. That is, if your largest client pays you $2 million per year, and total revenue is $20 million per year, then the concentration is 10%. Investors prefer companies with a low concentration ratio because a business that receives all its funds from only a few clients is more exposed to risk. For example, such customers may exert a bad influence on the product by demanding features that are only suitable for them, or force the company to sell them products and services at reduced prices.

LTV (Life Time Value) / Customer value throughout the life cycle

This metric allows you to find out the long-term value of a client - how much net profit you will receive from him (without acquisition costs) over the entire duration of your relationship. To calculate this value you need to know the following indicators.

1) Revenue per client (per month) = average cost of orders multiplied by the number of orders.

2) Customer contribution = customer revenue minus various customer costs (eg, sales, administrative and other service-related operating expenses).

3) Average customer lifetime (in months) = 1 divided by monthly churn rate.

To obtain the lifetime value indicator, the contribution towards the coverage must be multiplied by the average life expectancy of the client. The higher the indicator, the more promising the company is for the investor.

Burn Rate / Speed ​​of spending funds

Knowing your burn rate is especially important for early-stage startups. When money runs out, young companies have no room to cut costs and no time to increase investment. The formula for calculating the dangerous rate is simple: from the balance at the beginning of the year, you need to subtract the balance at the end of the year and divide the result by 12. A year is not a strict time limit; you can take any number of months and, accordingly, substitute it into the divisor.

CMGR (Compounded Monthly Growth Rate)

Monthly growth is very easy to measure - it is usually calculated as the average of all monthly indicators for the year. However, Andreessen Horowitz assures that investors are interested in the aggregate indicator, which is calculated using a more complex formula. It is as follows: (last month of the year / first month of the year) * (1 / Number of months) – 1. For small companies, this figure may be lower than just the average growth rate.

Churn / Customer Churn

It is important to understand not only how many clients stay with you, but how many leave. The formula for calculating outflow will help with this - there are several options. Andreessen Horowitz offers the simplest one: the number of lost clients should be divided by the number of clients at the beginning of the period. There is no need to take into account new transactions made during this month.

The most stringent requirements apply to cloud services running on a subscription basis - monthly churn should not exceed 5–7%. In other areas, this figure can rise above 20%, but of course, the lower, the better for the investor.

“Secret” thanks Oleg Seidak, CEO of Blackmoon Financial Group, for his help in preparing the material