Realtime Compute for Apache Flink:Agrégation analytique

Modifications compatibles

• Ajoutez, supprimez ou modifiez un champ agrégé non distinct. Un champ agrégé correspond au résultat renvoyé par une fonction d'agrégation.

  • Partiellement compatible : ajoutez un champ agrégé. La valeur du nouveau champ est incrémentée au démarrage du job.

  • Entièrement compatible : supprimez un champ agrégé existant. Les données d'état associées sont ignorées.

  • Partiellement compatible : ajoutez un champ agrégé tout en supprimant un champ existant. La valeur du nouveau champ est incrémentée au démarrage du job. Les données d'état du champ supprimé sont ignorées.

  • Partiellement compatible : modifiez un champ agrégé existant. Cette opération équivaut à supprimer l'ancien champ et à en ajouter un nouveau. La valeur du nouveau champ est incrémentée au démarrage du job. Les données d'état de l'ancien champ sont ignorées.

  Remarque

  • Si vous ne modifiez pas un champ agrégé, les résultats de calcul restent identiques, avec ou sans utilisation des données d'état.

  • Une requête d'agrégation analytique renvoie les valeurs agrégées et les données d'entrée correspondantes. Toute modification du schéma d'entrée rend le job incompatible avec les données d'état.

  -- Original SQL statement: 
  select a, b, c,
    sum(b) over (partition by a order by ts),
    max(c) over (partition by a order by ts)
  from MyTable;
  
  -- Partially compatible: Add count(c), which is an aggregated field. 
  -- The calculation results of sum(b) and max(c) are not affected. The value of count(c) is incremented from 0 when the job starts. 
  select a, b, c,
    sum(b) over (partition by a order by ts),
    max(c) over (partition by a order by ts),
    count(c) over (partition by a order by ts)
  from MyTable;
  
  -- Fully compatible: Delete sum(b). 
  -- The calculation results of max(c) are not affected. 
  select a, b, c,
    max(c) over (partition by a order by ts)
  from MyTable;
  
  -- Partially compatible: Modify an aggregated field from max(c) to min(c). 
  -- The calculation results of sum(b) are not affected. The max(c) field is considered deleted and its state data is discarded. 
  -- The min(c) field is considered as a new field and its value is incremented when the job starts. 
  select a, b, c,
    sum(b) over (partition by a order by ts),
    min(c) over (partition by a order by ts)
  from MyTable;

• Entièrement compatible : modifiez l'ordre des champs agrégés non distincts.

  -- Original SQL statement: 
  select a, b, c,
    sum(b) over (partition by a order by ts),
    max(c) over (partition by a order by ts)
  from MyTable
  
  -- Fully compatible: Modify the sequence of sum(b) and max(c). 
  -- The calculation results of sum(b) and max(c) are not affected. 
  select a, b, c,
    max(c) over (partition by a order by ts),
    sum(b) over (partition by a order by ts)
  from MyTable;

• Entièrement compatible : modifiez l'ordre des clés de partitionnement.

  -- Original SQL statement: 
  select a, b, c,
    sum(b) over (partition by a,b order by ts),
    max(c) over (partition by a,b order by ts) 
  from MyTable;
  
  -- Fully compatible: Modify the sequence of partition keys. 
  select a, b, c,
    sum(b) over (partition by b,a order by ts),
    max(c) over (partition by b,a order by ts) 
  from MyTable;

• Entièrement compatible : modifiez l'ordre des champs dans le schéma ou ajoutez/supprimez des champs sans altérer le schéma.

  -- Original SQL statement: 
  select a, b, c, count(b) 
  over (partition by a order by ts) from MyTable;
  
  -- Fully compatible: Modify the sequence of fields b and c for the OVER clause. 
  select a, c, b, count(b) 
  over (partition by a order by ts) from MyTable;
  
  -- Original SQL statement: 
  select a, b, c, cnt from (select a, b, c, d, count(b) 
  over (partition by a order by proctime) as cnt from src);
  
  -- Fully compatible: Delete a field that is not used for aggregation. 
  select a, b, c, cnt from (select a, b, c, count(b) 
  over (partition by a order by proctime) as cnt from src);

Modifications incompatibles

• Modifiez le schéma d'entrée d'une requête d'agrégation analytique.

  -- Original SQL statement: 
  select a, b, c,
    sum(b) over (partition by a order by ts),
    max(c) over (partition by a order by ts)
  from MyTable;
  
  -- Incompatible: Add d as an input field. 
  select a, b, c, d,
    max(c) over (partition by a order by ts),
    sum(b) over (partition by a order by ts)
  from MyTable;
  
  -- Incompatible: Modify the input field c. 
  select a, b, c,
    max(c) over (partition by a order by ts),
    sum(b) over (partition by a order by ts)
  from (
    select a, b, substring(c, 1, 5) as c from MyTable
  );

• Modifiez un attribut spécifié dans la clause OVER, comme les clés de partitionnement, l'ordre de tri ou le cadre de la fenêtre.

  -- Original SQL statement: 
  select a, b, c,
    max(c) over (partition by a order by ts asc rows between unbounded preceding and current row)
  from MyTable;
  
  -- Incompatible: Change the partition key from a to b. 
  select a, b, c,
    max(c) over (partition by b order by ts asc rows between unbounded preceding and current row)
  from MyTable;
  
  -- Incompatible: Change the sorting order from ascending to descending. 
  select a, b, c,
    max(c) over (partition by a order by ts desc rows between unbounded preceding and current row)
  from MyTable;
  
  -- Incompatible: Change the bound definition from unbounded preceding to 2 preceding. 
  select a, b, c,
    max(c) over (partition by a order by ts asc rows between 2 preceding and current row)
  from MyTable;

• Ajoutez, supprimez ou modifiez un champ agrégé distinct.

  -- Original SQL statement: 
  select a, b, c,
    max(c) over (partition by a order by ts)
  from MyTable;
  
  -- Incompatible: Add count(distinct b), which is a distinct aggregated field. 
  select a, b, c,
    max(c) over (partition by b order by ts),
    count(distinct b) over (partition by b order by ts)
  from MyTable;

• Modifiez tous les champs agrégés.

  -- Original SQL statement: 
  select a, ts, min(b) over (partition by a order by ts) 
  from MyTable;
  
  -- Incompatible: Modify the only aggregated field. 
  select a, ts, max(b) over (partition by a order by ts) 
  from MyTable;

Modifications dont la compatibilité est inconnue

L'utilisation d'une fonction d'agrégation définie par l'utilisateur (UDAF) Python avant ou après la modification empêche le système de déterminer la compatibilité.

-- Unknown compatibility: Use Python UDAFs, such as the weighted_avg function shown in the following code, before or after the modification. 
select min(a), min(b), min(c), weighted_avg(a, b), min(cnt) 
from (select a, b, c, count(b) 
over (partition by a, b order by ts) as cnt from MyTable);