This topic describes the mappings of data and value types between Spark, Scala, as well as the search indexes and tables of Tablestore. When you use these data and value types, you must follow the mapping rules for Spark, Scala, and Tablestore.
Basic data types
The following table describes the supported basic data types.
Data type in Spark | Value type in Scala | Data type in search indexes | Data type in tables |
---|---|---|---|
ByteType | Byte | Long | Integer |
ShortType | Short | Long | Integer |
IntegerType | Int | Long | Integer |
LongType | Long | Long | Integer |
FloatType | Float | Double | Double |
DoubleType | Double | Double | Double |
StringType | String | Keyword/Text | String |
BinaryType | Array[Byte] | Binary | Binary |
BooleanType | Boolean | Boolean | Boolean |
String JSON (geographical coordinates) | String (JSON) | Geopoint | String (JSON) |
Geographical location data types
Search indexes support geo query. When a geo query is pushed down to the compute layer, Spark can query and analyze data based on geographical locations in addition to basic types of data.
Geo query provides the following query types: geo-distance query, geo-bounding box query, and geo-polygon query. Geo query can be used to query information about the locations of IoT devices, locations of food delivery orders, locations of employees when they clock in or out, and locations of express delivery orders. You can use the following methods to perform geo query:
Use the search indexes of Tablestore. For more information, see Geo-distance query.
Use Spark SQL.
Use geo-distance query by specifying the central point and the radius for the query.
select * from table where val_geo = '{"centerPoint":"3,0", "distanceInMeter": 100000}' and name like 'ali%'
Use geo-bounding box query.
select * from table where geo = '{"topLeft":"8,0", "bottomRight": "0,10"}' and id in { 123 , 321 }
Use geo-polygon query.
select * from table where geo = '{"points":["5,0", "5,1", "6,1", "6,10"]}'
The following table describes the geographical location data types supported by Spark, Scala, search indexes, and tables.
Data type in Spark | Value type in Scala | Data type in search indexes | Data type in tables |
---|---|---|---|
String JSON (coordinates of the central point and the radius of a circular geographical area) | String (JSON) | Geopoint | STRING (JSON) |
STRING JSON (coordinates of the vertices in a rectangular geographical area) | String (JSON) | Geopoint | STRING (JSON) |
String JSON (coordinates of the vertices in a polygonal geographical area) | String (JSON) | Geopoint | STRING (JSON) |