Predicate Calculus as a Tool for AI Problems Solution: Algorithms and Their Complexity Predicate Calculus as a Tool for AI Problems Solution: Algorithms and Their Complexity

The chapter is devoted to the use of predicate calculus for artificial intelligence (AI) problem solving. Here, an investigated object is represented as a set of its elements and is characterized by a fixed number of predicates. Its description is a set of all constant literals (with the chosen predicates), which are valid on the object. The NP-complete problem, “ whether an object satisfies a goal formula, ” is under consideration. The upper bound of number of its solution steps is exponential. The notion of common up to the names of arguments subformula of two predicate formulas and one of their isomorphisms allows to construct a level description of the set of goal formulas and essentially to decrease the upper bounds of the problem solving. The level description permits to define a self- training predicate network, which may change its configuration during the process of training. The extraction of common up to the names of arguments subformulas permits to construct a multiagent description of an object when everyagent does not know the true number of the object elements and uses her own notifications for the names of elements. A model example illustrating all algorithms is presented.


Introduction
The choice of initial attributes for description of an object in an artificial intelligence (AI) problem is the first stage of any simulation of an informational process (representation of information for its further use).
At the 60-70th of the twentieth century, many authors (see, for example, [1]) offered to use predicate calculus for AI problem solving. The resolution method seemed to be a very easy and clear tool to solve problems dealing with compound objects, which can be described by properties of its elements and relations between these elements.
Until the notion of NP-complete problem (in particular, described in [2,3]) was not widely adopted, such an approach seemed to be very convenient, but many such-a-way formalized problems occurred to be NP-complete or even algorithmic unsolvable.
While developing the effective algorithms deciding discrete problems, determination of estimations for number of steps of their run becomes one of the important problems. The absence of the proved estimations for number of an algorithm run steps is considered as an insufficient research of this algorithm. It is especially relevant for problems with big input. It concerns, in particular, to the algorithms deciding various AI problems. At practical use of an algorithm, it is important that it has polynomial upper bound of number of its run steps. The NPcompleteness or NP-hardness of a problem means now that the polynomial algorithm of its decision is not known.
In 2007, the author proved NP-completeness of a series of AI problems formalized with the help of predicate calculus formulas [4], proved upper bounds for number of steps of algorithms solving these problems [5], and offered a level description of goal formulas for decreasing the number of proof steps [6]. Such a level description is based on the extraction of a common up to the names of its arguments sub-formula of the set of elementary conjunctions of atomic predicate formulas. These sub-formulas define generalized characteristics of an object.
Extraction of such sub-formulas allows to construct logic-predicate networks [7], which may change its configuration (the number of layers and the number of cells in the layer) during the process of training.
Extraction of these sub-formulas may serve as an instrument for constructing a multiagent description of an object, when every agent can describe only a part of the object (these parts are intersected), but every agent gives its own names to the elements of the whole object [8].
Here, some AI problems formalized in such a way are under consideration. For these problems, the solving algorithms and upper bounds of their run are obtained. These upper bounds permit to point out the parameters of the problem, which mostly influence on the complexity of the algorithm, and to offer approaches permitting to decrease the complexity.
A model example illustrating the described approach and algorithms is given.

Logic-predicate approach to some AI problems and number of steps of these problems solution
Let an investigated object be presented as a set of its elements ω = {ω 1 ,…, ω t }. The set of predicates p 1 ,…, p n (every of which is defined on the elements of ω) characterizes properties of these elements or relations between them. Logical description S(ω) of an object ω is a collection of all true formulas in the form p i τ ð Þ or ¬p i τ ð Þ (where τ is an ordered subset of ω) describing the properties of ω elements or relations between them.
Let the set Ω of all investigated objects be a union of classes Ω k , (k = 1,…, K), i.e., Ω = ⋃ K k¼1 Ω k . Logical description of the class Ω k is such a formula A k x ð Þ that if the formula A k ω ð Þ is true then ω ∈ Ω k . The class description may be represented as a disjunction of elementary conjunctions of atomic formulas.
Here and below, the notation x is used for an ordered list of the set x. To denote that there exists such a list x that all values for variables from the list x are distinct the notation ∃x 6 ¼ A k x ð Þ is used.
The introduced descriptions allow to solve many artificial intelligence problems [9]. Main of these problems may be formulated as follows.
Identification problem: to pick out all parts of the object ω that belongs to the class Ω k .
Classification problem: to find all such class numbers k that ω ∈ Ω k .
Analysis problem: to find and classify all parts τ of the object ω.
The solution of these problems may be reduced to the proof of logic sequents respectively, and determination of the values for x and k. The number of A k x ð Þ variables in the sequent (2) must be equal to the number of constants in ω.
Note that the proof of any of the sequent (1), (2), or (3) answers only the question "whether it is true?" Strictly speaking, in the sequents (1)-(3), instead of the symbols ∃x and ⋁ K k¼1 there must be words "what are the distinct values of x" denoted as ?x ð Þ and "what are the values of k?" denoted as ? K k¼1 , respectively. In such a case, the sequents (1)-(3) would take the form If one uses an exhaustive or a logical algorithm (derivation in a sequent calculus or proof by resolution method), the algorithm gives the values for x and k.
The proof of sequents (1) and (3) is based on the proof of the sequent where A x ð Þ is an elementary conjunction. It follows from the fact that A k x ð Þ is a disjunction of the form C 1 ∨…∨C r of elementary conjunctions of atomic formulas C 1 ,…,C r , and ∃x 6 ¼ (C 1 ∨…∨C r ) ⇔ (∃x 6 ¼ C 1 ∨…∨ ∃x 6 ¼ C r ). That is why we can consecutively check the sequents of the form S ω ð Þ ) ∃x 6 ¼ C j . Here, the maximal value for j is r for the sequent (1), and the sum of the number of elementary conjunctions in all class descriptions for the sequent (3).
An exhaustive algorithm is widespread to prove (4). The total estimate for the number of steps (i.e., the number of comparisons) for the exhaustive algorithm solving (4) is or, more roughly, O t m as ð Þ: While using a logical algorithm (derivation in a predicate sequent calculus or proof by resolution method for predicate calculus), one must find unifier of the formula A x ð Þ and some subset of S(ω).
The number of steps (i.e., the number of comparisons) required for the solution of the system and, hence, for the logical algorithm solving (4) is a i and s i be the numbers of literals with the predicate p i in A x ð Þ and in S(ω), respectively. More roughly where s 0 = max{s 1 ,…, s n }.
The above-received estimations are exponential over the length of A x ð Þ. The ones for an exhaustive algorithm are exponential over the number of variables, and for a logical algorithm they are exponential over the maximal number of literals with the same predicate. It allows to choose the algorithm depending on characteristics of the concrete problem under consideration. Note that the reverse Maslov's method [10,11] has the same estimations for the solution of the sequent (4), but makes essentially smaller number of steps on the average. The received estimations cannot be essentially decreased up to polynomial ones if P 6 ¼ NP (classes P and NP are the classes of predicates checked in polynomial time by a deterministic or nondeterministic Turing machine respectively). More precisely, the problem (4) is NP-complete and, hence, the problems (1) and (3) are NP-complete, and the problems (4) and (5) are NPhard [4,5].
Problem (2) is strictly connected with the so-called "open" problem ISOMORPHISM OF GRAPHS [3], for which it is not proved neither its polynomiality nor its NP-completeness.
Given a "box" inside a complex contour image containing t nodes and s be the maximal number of occurrences of the predicate V in the description S(ω), it would be recognized (according to the estimations (5 0 ) and (6 0 )) in O(t 10 ) steps by an exhaustive algorithm and in O (s 37 ) steps by a logical algorithm.
Can seem that many atomic formulas such as V( are unnecessary. But if we delete such "unnecessary" formulas, it would be needed to add to a premise of a sequent, a condition that every point that belongs to a segment (y,z) may be substituted instead of y or z (be the second or the third argument) in every atomic formula with the predicate V. It would be another setting of a problem. Moreover, such "unnecessary" formulas can help to decrease the number of algorithm run steps if we use branch and bound algorithm inside the exhaustive algorithm or the reverse Maslov's method for a logical one [11].

Level description of classes
Below, the designation A k x k ð Þ will be used for elementary conjunctions, which are disjunctive terms of a class description.
The notion of level description of classes was introduced in [6]. Such a description essentially allows to decrease the number of steps for an algorithm solving every of the above-formulated problems. This notion is based on the extraction of "frequently" appeared "sub-formulas" small complexity" and changing them in these formulas by atomic formulas p 1 i y 1 i À Á defined by an equivalence of the form New predicates p 1 i having new first-level arguments y 1 i for lists y 1 i of initial variables are called first-level predicates. The formula After L repetitions, an L-level description in the following form is received: The solution of the problem of the form (4) with the use of the level description of classes is decomposed on the sequential (l = 1, …, L) implementation of the actions 1-4: 1. For every i (i = 1,…, n l ) check S lÀ1 (ω) ) ∃y 1 i 6 ¼P 1 i y 1 i À Á and find all lists τ 1 i of previous levels constants for the values of the variable list y 1 i such that S lÀ1 (ω) )P 1 4. Add all constant atomic l-level formulas in the form p 1 i τ l i À Á (τ 1 i were received at the first step) to S lÀ1 (ω) and obtain S l (ω) . Here τ l i τ i l are new l-level constants for the lists of (l À 1)-level constants The decreasing of the number of steps for an algorithm solving every of the above formulated problems (1)-(3) with the use of a level description follows from the fact that in items 1, 2, and 5, we solve the same problem as it was formulated in Section 1 and has the number (4). The estimations of number of steps exponentially depend on the parameters of the formula, i.e., on the right part of implication. That is why the term "small complexity" for P l i y l i À Á must be interpreted as "small number of variables in P l i y l i À Á " for an exhaustive algorithm, and "small number of literals in P l i y l i À Á " for a logical algorithm Why did we use quotation marks for the term "sub-formulas?" Such formulas (elementary conjunctions) P l j y l j are not obliged to be precisely sub-formulas of A 1 x 1 ð Þ, …, A K x K ð Þ but may differ from these sub-formulas in names of variables and order of conjunctive terms Definition 1. Elementary conjunctions P and Q are called isomorphic if there is an elementary conjunction R and substitutions λ R,P and λ R,Q of the arguments of P and Q, respectively, instead of the variables in R such that the results of these substitutions coincide up to the order of literals The substitutions λ R,P and λ R,Q are called unifiers of R with P and Q, respectively.

Definition 2.
Elementary conjunction C is called a common up to the names of arguments sub-formula of two elementary conjunctions A and B if it is isomorphic to some sub-formulas A 0 and B 0 of A and B, respectively The formula P(u,v) is their common up to the names of variables sub-formula with the unifiers λ P,A -substitution of x and y instead of u and v, respectively, and λ P,B -substitution of x and z instead of u and v, respectively. It is so because An algorithm of extraction of a maximal (having a maximal number of literals) common up to the names of arguments sub-formula C of two elementary conjunctions A and B and determining the unifiers λ c,A 0 and λ c,B 0 is described in [12]. The number of steps of this algorithm is O(a a b b ), where a and b are the numbers of literals in A and B, respectively. The minimal number of steps of this algorithm is O((ab) 2 ), the middle estimate is O((ab) 1/2 log(ab) ).
This algorithm allows to construct a level description for a set of goal elementary conjunctions. Essential difference between maximal common up to the names of arguments sub-formulas and sub-formulas in the level description consists in the fact that in the level description it is needed to extract sub-formulas with "small complexity" but not a maximal one. An algorithm of level description construction is in [6]. It consists in sequential pairwise extraction of common up to the names of variables sub-formulas of A i x i ð Þ and A j x j À Á with finding their unifiers and then the analogous procedure with the obtained sub-formulas.
Let N be the maximal number of literals A k x k ð Þ in (k = 1,…, K). The upper bound of this algorithm number of steps is O(K 2 N 2N ).

Example of sub-formula extraction and a level description construction
Return to the example in the previous section. There, we have seen a description of a class of "boxes" represented in Figure 1. According to these descriptions, we have received that given a "box" inside a complex contour image containing t nodes it would be recognized in O(t 10 ) steps by an exhaustive algorithm and in O(s 37 ) steps by a logical algorithm (here, s is the maximal number of occurrences of the same predicate in the object description S(ω)).
Pairwise extraction of common up to the names of variables of elementary conjunctions, corresponding to these images, allows to extract common up to the names of variables subformulas corresponding to the images represented in Figure 2 These sub-formulas contain, respectively, 8, 8, 7, 7, 7, 8 variables and 18, 15, 11, 11, 15, 16 atomic formulas.
The following extraction by means of pairwise partial deduction between common subformulas corresponding to images ab, ac, ad, bc, bd, cd gives a sub-formula corresponding to the image represented in Figure 3.  5 ), corresponding to this image, defines a first-level predicate p 1 (x 1 ). The first-level variable x 1 is a variable for a list of seven initial variables x 1 = (x 1 ,x 2 ,x 3 ,x 4 ,x 5 ,x 9 ,x 10 ). The unifier of P 1 (x 1 , x 2 ,x 3 ,x 4 ,x 5 ,x 9, x 10 ) with the description of ab, ac, ad, ac, and bd is an identical substitution, but its unifier with the description of cd is a substitution of x 6 instead of x 5 .

Logic-predicate network
Traditional neuron network deals with binary or many-valued characteristics of an object and is an adder of weighted inputs followed by a function mapping the result into the segment [0, 1]. The neuron network configuration is fixed and only the weights may be changed.
A logic-predicate network is described later. The inputs for this network are atomic formulas setting properties of the elements composing an investigated object and relations between them [7]. The proposed model of logic-predicate network has two blocks: a training block and a recognition block. The input of every block is an elementary conjunction of atomic predicate formulas or their negations. Configuration of the recognition block is formed after an implementation of the training block and may be changed with its help.
The training block is a "slowly running" block. At the same time, the recognition block is a "quickly running" one. The base of the proposed predicate network is a logic-objective approach to AI problems and level description of classes.
The scheme of the logic-predicate network is presented in Figure 4 At a training stage of logic-predicate network construction, we have a training set of objects. Let a training set of objects ω 1 ,…, ω K be given to form an initial variant of the network training block. Replace every constant ω j k in S(ω k ) by a variable x j k (k = 1,…, K, j = 1,…, t k ) and substitute the sign & between the atomic formulas. Initial goal formulas A 1 x 1 ð Þ, …, A K x K ð Þ are obtained.
Construct a level description for these goal formulas with the use of algorithm of level description. The first approximation to the recognition block is formed. Formulas P l i y l i À Á (i = 1,…, n l , l = 1,…, L) obtained in the training block (together with the unifiers) are the contents of the cells forming the recognition block. This block runs as it was described in the section level description of classes.
The recognition block tries to identify a new object according to the level description of classes, obtained in the training block.
If after the "recognition block" run an object is not recognized or has wrong classification, then it is possible to train anew the network. The description of the "wrong" object must be  added to the input set of the training block. The training block extracts common subformulas of this description and previously received formulas forming the recognition block. Some sub-formulas in the level description would be changed. Then, the recognition block is reconstructed.

Model example of a logic-predicate network construction
Given a training set for the class of contour images of "boxes" presented in Figure 1 (Section 2). Pairwise extraction of common up to the names of variables of elementary conjunctions, corresponding to these images, allows to extract common sub-formulas corresponding to the images presented in Figures 2 and 3 (Section 3). Fragments of the images corresponding to a three-level network are presented in Figure 5. Given, a new image represented in Figure 6 for recognition, the network would not recognize it because the first-level predicate is not valid.
Add the description of this control image to the input data of the training block. The extraction of common sub-formulas for this description and the formula defining the first-level predicate gives a formula corresponding to the image represented in Figure 7.
New second-level predicates correspond to three images represented in Figure 8.
The set of the third-level predicates coincides with the set of previous second-level predicates. So, the recognition block is constructed anew and represents four-level description of the class. Fragments of the images corresponding to a four-level network are presented in Figure 9.

Multi-agent description of an object
A problem of multi-agent description of a complex object is under consideration in this section. It is supposed that every agent knows only a part of an investigated object description. Moreover, she does not know the true names of elements and gives them names arbitrary. It is similar to the parable about tree blind men who feel an elephant. To overcome such a   Predicate Calculus as a Tool for AI Problems Solution: Algorithms and Their Complexity http://dx.doi.org/10.5772/intechopen.72765 Figure 9. Fragments of the images corresponding to a four-level network.
paradox, it is supposed that every two agents have information concerning some common part of an object. The main difficulty in this problem is to find and identify these parts [8].

Setting of the problem
Let an investigated object is represented as a set of its elements ω ¼ ω 1 ; …; ω t f gand is characterized by the set of predicates p 1 ,…, p n , every of which is defined on the elements of ω and gives properties of these elements or relations between them.
Information (description) of an object is an elementary conjunction of atomic formulas with predicates p 1 ,…, p n and some constants as arguments.
There are m agents a 1 ,…, a m which can measure some values for some predicates of some elements of ω. The agent a j does not know the true number of the ω elements and suppose that she deals with the object ω j ¼ ω As every agent uses her own notifications for the names of the object elements, it is needed to find all common up to the names of arguments sub-formulas C ij of the information and their unifiers, i.e., such substitutions for the argument names that the extracted pairs of sub-formulas are identical.

Algorithm of multi-agent description
Below, the arguments of information will be omitted. Let every agent a j has information I j about the described object ω (j = 1,…, m). To construct a description of ω the following algorithm is offered.
1. Change all constants in I 1 ,…, I m by variables in such a way that different constants are changed by different variables and the names of variables in Ii and Ij (i 6 ¼ j) does not coincide. Obtain I 0 1 ,…, I 0 m .

2.
For every pair of elementary conjunctions I 0 i and I 0 j (i = 1,…, m À 1, j = i + 1, …, m) find their maximal common up to the names of arguments sub-formula C ij and unifiers λ i,ij and λ j,ij . Every argument of C ij has a unique name.

3.
For every pair i and j (i > j) check if I 0 i and I 0 j contain a contradictory pair of atomic formulas or two sub-formulas which cannot be satisfied simultaneously (for example, "x is green" and "x is red"). If such a contradiction is established, then delete from C ij atomic formulas containing the variables, which are in the contradictory sub-formulas. Change the unifiers by means of elimination of these variables.

4.
For every i, identify the variables in C ij (i 6 ¼ j) which are substituted in I 0 i and I 0 j instead of the same variable. The names of the identified variables are changed in unifiers by the same name.

5.
With the use of the unifiers obtained in items 2-4 change the names of variables in I 0 1 ,…, I 0 m . Obtain I 00 1 , …, I 00 m .
6. Write down the conjunction I 00 1 & … & I 00 m and delete the repeated atomic formulas.

Upper bound of the number of steps
To estimate the number of the algorithm run steps, we estimate every item of the algorithm. "steps" for an algorithm based on the derivation in the predicate calculus.
It is needed to summarize the above estimates for i = 1,…, m À 1, j = i, …, m. So, we have O t t •2 s m 2 À Á "steps" for an exhaustive algorithm and O s sþ3 •m 2 À Á "steps" for an algorithm based on the derivation in the predicate calculus. Here, t and I k k are the maximal numbers of variables and atomic formulas in I j (j = 1,…, m), respectively.
3. Consistency checking of the formulas I i and I j requires I i k k I j "steps." This item of the algorithm requires not more than P m i¼1 m À i ð Þ I i k k "steps" that is O(m 2 s) "steps." 4. For every i, identification of the variables in C ij (i > j) consists in the comparison of the replaced part of the unifiers λ i,ij and λ j,ij . It requires not more than (m À i)t i 2 "steps." Summarizing it for i = 1,…, m we have not more than 5. The number of "steps" required for the changing of the names of variables in I1, …, Im is linear under 6. The number of "steps" required for the deleting of the repeated conjunctive terms is not more than P mÀ1 i¼1 P m j¼iþ1 I i k k I j "steps." The whole number of the algorithm run steps is O(t t 2 s m 2 ) for an exhaustive algorithm and O (s s + 3 m 2 ) for an algorithm based on the derivation in the predicate calculus.
The analysis of the received estimation shows that the main contribution is made by the summarized number of partial deduction checking (item 2).

Example of a multi-agent description
Let the initial predicates be V and L described in Section 2. Each of the three agents has a description of one of the fragment presented in Figure 10.
As a result, we have the following descriptions of the fragments: Their conjunction allows to "stick together" the images of fragments according to the same variable. The image corresponding to the result of "sticking" is presented in Figure 11.
If a description of the investigated object is presented in the database, it may be found according the principle "the nearest neighbor" with the use of metric for predicate formulas presented in [13]. Figure 11. Image corresponding to the result of "sticking".
Predicate Calculus as a Tool for AI Problems Solution: Algorithms and Their Complexity http://dx.doi.org/10.5772/intechopen.72765

Conclusion
Logic-predicate approach to an AI problem has a rather powerful capability, essentially when an investigated object is a compound one and is characterized by properties of its elements and relations between them.
Setting of pattern recognition problems considered in Section 2 (except the problem (2)) differs from the classical one. The setting of the problems (1) and (3), in which it is needed to find parts of an investigated object, turns out to be a rather difficult one in the frameworks of a standard approach in the frameworks of which an object is regarded as a whole indivisible one.
In particular, an exponential estimation for number of propositional variables in a formula simulating a predicate formula in a finite domain for planning problems TÁ|Act|ÁO P is mentioned in [14]. Here, T is the number of time stages, |Act| is the number of schemes of actions, O is the number of objects in the domain, P is the maximal number of parameters in schemes of actions. In Section 2, the analogous estimate (5 0 ) was received for an exhaustive algorithm solving the problem (4).
The problem (2) is polynomial equivalent to an "open" problem ISOMORPHISM OF GRAPHS [3] and the problems (1) and (3) are NP-complete.
A notion of level description of classes has been introduced in Section 3 in order to decrease the number of steps of algorithms solving these problems. Such a description reduces the solution of the main problem to a series of solutions of the same form problems with the inputs with the essentially less notation lengths. At the same time, the constructing of a level description still deals with big input data. So, a problem with big input data is solving only once, and then the problem with the essentially less input data is solving repeatedly.
The idea of decomposition of a problem to a series of the "less dimension" problems is not a new one and is frequently used. The difficulty consists in a precise definition of the term "common sub-formula of small complexity." The development of a precise definition and of an algorithm for the extraction of a common up to the names of arguments sub-formula of two elementary conjunctions (and their unifiers) allows not only to work out an algorithm of level description construction but also to find an approach to the solution of some else AI problems.
Note that the extracted sub-formulas define generalized characteristics of an object. This has an analogy in medical diagnostics: initial characteristics are symptoms and the generalized ones are syndromes.
Level description of classes allowed to introduce the notion of logic-predicate network described in Section 4. Such a network may be regarded as a self-training network which changes its configuration after an additional training. It corresponds to the fact that in the process of a man training, new notions and relations between them are formed in a human brain.
The presence of an algorithm for the extraction of a common up to the names of arguments sub-formula of two elementary conjunctions (and their unifiers) allows to find an approach to a problem of multi-agent description of an object described in Section 5. Just an extraction of such sub-formulas and determining of their unifiers with the input formulas makes possible to "stick together" such parts of descriptions in which different agents gives different names to one element of the whole object.
Note that the formulation of the problem (1) from Section 2 coincides with the one for a wellknown problem CONJUNCTIVE BOOLEAN QUERY from [3]. The difference is in the implementation of these problems. While repeated implementation of the problem (1) the premise S(ω) of the sequent S ω ð Þ ) ∃x 6 ¼ A k x ð Þ is different, while every implementation and the conclusion A k x ð Þ is a constant part. That is why just a collection of class descriptions A 1 x 1 ð Þ, …, A K x K ð Þ permits to construct a level description.
While repeated implementation of the problem CONJUNCTIVE BOOLEAN QUERY, the premise S(ω) of the sequent S ω ð Þ ) ∃x 6 ¼ A k x ð Þ is constant while every implementation and queries A k x ð Þ are different while every implementation. An approach to the construction of a level data base is presented in [15].
The possibility of reduction of an object description length by means of adding a formula setting some properties of initial predicates to the premise of a sequent was mentioned in the model example in Section 2. Properties of initial predicates also were used in the item 3 of the algorithm of multi-agent description. In fact, in the both cases instead the sequent of the form (4) S ω ð Þ ) ∃x 6 ¼ A x ð Þ it is needed to check another sequent of the form C y ð Þ S ω ð Þ ) ∃x 6 ¼ A x ð Þ, where C y ð Þ is a set of formulas setting properties of initial predicates. Investigation of computational complexity of such a form sequent may be an interesting problem in the further research.
To solve the problem (2) and to extract a maximal common up to the names of arguments subformula of two elementary conjunctions it is needed to check whether two elementary conjunctions are isomorphic. A polynomial in time rough algorithm for such a checking was offered in [12] by Petrov. Numerical experiments with this algorithm give over 99.95% of valid results.