INVENTION | FREE FULL TEXT | Identification of two-phase flow patterns in vertical pipes using transformer neural networks

Transformer’s architecture was originally designed for sequence-to-sequence tasks such as machine translation, but independent encoders and decoders were quickly adopted, resulting in three different types of Transformers [32]:

–

Encoder only: This model can convert text input sequences into numerical representations suitable for tasks such as text classification or named entity recognition. The representation computed for a given token in this architecture depends on bidirectional attention, meaning it relies on the left and right context of the token.

–

Decoder only: This type of model is able to automatically complete a sequence by iteratively predicting the most likely next word. In this case, the model relies on causal or autoregressive attention, which means that the representation computed for a token in this architecture only depends on the left-hand context.

–

Encoder-Decoder: They are used to model complex mappings from one text sequence to another, making them suitable for tasks such as machine translation and summarization.

For this work, the encoder-only model is used. Therefore, the structure of the encoder block is explained in detail below. The structure of the encoder block is shown in Figure 2. The embedding sequence enters this block and is processed through the following layers: multi-head self-attention layer and feed-forward layer.

Self-attention is a mechanism that allows a neural network to assign different attention, that is, different weight values, to each element of a sequence. Therefore, the main idea of ​​self-attention is to use the entire sequence to calculate the weighted average of each embedding.One way to express this is that given a sequence of tokens $X_1$…, $X_n$generating a series of new embeddings from attention ${X_1}^{‘}$…, ${X_n}^{‘}$each of which ${X_I}^{‘}$ are all linear combinations $X_j$as shown in equation (1):

where the coefficient $w_{jI}$ are the attention weights and are normalized in this way: $S_j{w}_{jI}=1$.

The self-attention process involves three vectors: $\mathrm{ask}$,Inquire, $K$, key y, and $V$, valuesobtained by multiplying each input entry by the weight matrix watt (Trained during training).Calculate the dot product between vectors $\mathrm{ask}$ and vector $K$. The result is then divided by the square root of the dimensionality of the original input vector ($d_k$), use normalized soft max operation and multiply the vector V. For computational efficiency, information is processed in matrix form. Equation (2) represents the attention process.

The database was constructed based on existing experimental data from the literature related to two-phase flow of water and oil in vertical pipes. A total of 4864 data points were collected from 18 different authors, as shown in Table 1, which provides comprehensive information about each author’s data.Shown in the table are pipe diameters $\left(D\right)$ Viscosity used in experiments $\left({\mathrm{rice}}_{\mathrm{oh}}\right)$ and density $\left({\mathsf{r}}_{\mathrm{oh}}\right)$ The type of oil used, the number of flow patterns identified, and the amount of data extracted.

Based on the protocol developed in each study, a total of 9 different flow patterns were identified, characterized by changes in the continuous phase, either water (w) or oil (o). These modes include very fine dispersed droplets (VFD) o/w (265 cases) and w/o (204 cases), droplet (D) o/w (947 cases) and w/o (1292 cases), segment Plug (S) o/w (459 cases) and w/o (656 cases), drain (21 cases) o/w and (126 cases) w/o, annular (core flow) 480 cases, and transition region (TF ) 314 cases. Figure 3 provides a graphical representation illustrating the behavior of the most prominent flow modes.

The structure of the implemented model consists of the encoder implemented in the TNN and the SoftMax layer at the model output.input vector $X$ Consists of the apparent velocities of the fluids (oil and water), their sum, the in-situ volume fractions of the two fluids (hold-up volume), the viscosity of the oil and the diameter of the circular cross-section pipe, as shown in Equation (3) .

The training optimizer chosen is Adam because it combines the characteristics of two methods: one is the momentum gradient descent algorithm, and the other is the RMSP (Root Mean Square Propagation) algorithm. Gradient descent with momentum speeds up the gradient descent algorithm by considering an exponentially weighted average of the gradients, meaning that using the average allows the algorithm to converge to the minimum faster. The method is defined in equations (4) and (5).

Where

yes ${\mathrm{rice}}_t$ is the aggregated gradient at the current time step t, initially set to 0. also, ${\mathrm{rice}}_{t–1}$ Represents the aggregated gradient at the time step t – 1, ${\mathrm{oh}}_t$ is the weight at time step t, ${\mathrm{oh}}_{t+1}$ is the weight of the time step t + 1, $A$ is the learning rate at time step t. also, $dL$ is the derivative of the loss function, $d{\mathrm{oh}}_t$ is the derivative of the time step weight tand $\mathrm{Second}$ = 0.9 is the moving average parameter.

Instead of accumulating squared gradients, root mean square propagation (RMSP) uses an exponential moving average. Its definition is shown in the following formula (6).

Where

Where $v_t$ is the sum of squares of past gradients, that is, sum($dL$/${d\mathrm{oh}}_{t–1}$). initial, $v_t$ = 0. $\mathrm{Second}={10}^{–8}$ is a small positive constant to prevent division by zero $v_t⟶0$and $A=$ 0.001 is the learning rate parameter.

Therefore, Adam controls the gradient descent speed such that the global minimum is reached in small steps, and the steps are large enough to overcome local minima along the way, thereby achieving efficiency. In this case, equation (8) is used.

Where ${\mathrm{Second}}_1$ = 0.9 and, ${\mathrm{Second}}_2$ = 0.999 is the decay rate of the gradient average in the first two methods.

Whereas ${\mathrm{rice}}_t$ and $v_t$ Initialized to 0 according to the previous method, they tend to be biased towards 0 because ${\mathrm{Second}}_1$ and ${\mathrm{Second}}_2$ ≈ 1. Adam solved this problem by calculating the bias correction ${\mathrm{rice}}_t$ and $v_t$. This is done to control the weights when the global minimum is reached and avoid high oscillations near it, meaning it adapts to gradient descent in each iteration. The equations used are (9) and (10):

so

The loss function implemented in the training of the developed model is the binary cross-entropy function, which is usually used in binary classification problems, but can also be applied to problems where the predictor variable takes values ​​between 0 and 1. This function is defined as equation (11):

Where $y_I$ is the actual category to be predicted, $\widehat{y_I}$ is the predicted probability of the category, C is the number of classes, and n is the number of examples.

Variations using 4 different activation functions, where we have recover Function that converts an input value by zeroing out negative values ​​and leaving positive values ​​unchanged. It is defined as equation (12).

The second activation function is the sigmoid function, which is used in models that need to predict the probability of an outcome, since it takes a range of values ​​between 0 and 1. Its definition is shown in formula (13).

In addition, the hyperbolic tangent function is also implemented, its continuity is similar to the sigmoid function, but the output range is -1 to 1, and its definition is as shown in Equation (14):

Finally, the GeLU activation function is used, as defined in equation (15), which adds nonlinearity by multiplying a random sigmoid function by the input, similar to recover Function.

Furthermore, the models selected based on their performance generated a confusion matrix from which the precision and accuracy of each flow pattern was extracted. First, you need to calculate the values ​​of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN). The precision is defined by Equation (16) and the accuracy is defined by Equation (17).