Platform For AI:FeatureDB overview

VPC direct connection

FeatureDB provides VPC direct connection based on PrivateLink. After successful configuration, you can then use FeatureStore SDK in your VPC to access FeatureDB through a private connection based on PrivateLink, thereby improving read and write data performance and reducing access latency.

You can configure VPC direct connection through one of the following methods.

• Method 1: If you have no FeatureDB data store, click Create Source on the Store tab. When creating a FeatureDB data source, configure VPC, Zone and vSwitch, and Security Group Name in the VPC Direct Connection Configurations section. For specific instructions, see Online data store: FeatureDB.

• Method 2: If you have already created a FeatureDB data source, click feature_db on the Store tab. On the page that appears, click VPC Direct Connection Configurations. Specify VPC, Zone and vSwitch, and Security Group Name, and click OK.

Write features

For offline features, you can use FeatureStore Python SDK to run scheduled tasks through DataWorks to synchronize data from MaxCompute to FeatureDB.

       // Configure regionId, Alibaba Cloud account, FeatureStore project
       Configuration configuration = new Configuration("cn-beijing",
                Constants.accessId, Constants.accessKey,"fs_demo_featuredb" );

        // Configure FeatureDB username, password
        configuration.setUsername(Constants.username);
        configuration.setPassword(Constants.password);

        // If using public network to connect to FeatureStore, refer to the domain information above
        // If using VPC environment, no need to set
        //configuration.setDomain(Constants.host);

        ApiClient client = new ApiClient(configuration);

        // If using public network connection, set usePublicAddress = true, VPC environment does not need to set
        // FeatureStoreClient featureStoreClient = new FeatureStoreClient(client, Constants.usePublicAddress);
        FeatureStoreClient featureStoreClient = new FeatureStoreClient(client );

        Project project = featureStoreClient.getProject("fs_demo_featuredb");
        if (null == project) {
            throw  new RuntimeException("project not found");
        }

        FeatureView featureView = project.getFeatureView("user_test_2");
        if (null == featureView) {
            throw  new RuntimeException("featureview not found");
        }

        List<Map<String, Object>> writeData = new ArrayList<>();
        // Simulate constructing data to write 
        for (int i = 0; i < 10; i++) {
            Map<String, Object> data = new HashMap<>();
            data.put("user_id", i);
            data.put("string_field", String.format("test_%d", i));
            data.put("int32_field", i);
            data.put("int64_field", Long.valueOf(i));
            data.put("float_field", Float.valueOf(i));
            data.put("double_field", Double.valueOf(i));
            data.put("boolean_field", i % 2 == 0);
            writeData.add(data);
        }

        for (int i = 0; i < 100;i++) {
            featureView.writeFeatures(writeData);
        }

        // This only needs to be called once, if all data is written, ensure all writes are completed, after calling this interface, writeFeatures cannot be called again 
        featureView.writeFlush();

For real-time feature writing, the entire data row will be updated by default. If the written data only contains some fields, the unwritten fields will be set to empty. If you want to update only the written fields and merge them with the original data, you can make the following settings:

• Use Java SDK: Specify InsertMode.PartialFieldWrite.

  for (int i = 0; i < 100;i++) {
      featureView.writeFeatures(writeData, InsertMode.PartialFieldWrite);
  }

• Use Flink Connector: Set insert_mode to partial_field_write.

Read features

You can use FeatureStore SDK (Go/Java) or EasyRec Processor to read features.

FeatureStore SDK (Go/Java) supports KV point queries for offline/real-time features. By specifying the JoinID (primary key) value and feature name, you can complete key-value (KV) queries within milliseconds to obtain the target feature data. FeatureStore SDK (Go/Java) also supports KKV queries for behavior sequence features. By specifying the UserID value, you can query the assembled sequence feature results.

EasyRec Processor has integrated FeatureStore Cpp SDK, which supports pulling all feature data from FeatureDB into memory and supports millisecond-level polling to update real-time feature data to memory, thereby achieving higher performance reading.

Metrics

If you use FeatureDB as an online data source, after creating a feature view, click Data Monitoring on the right side of the target view to view metrics such as read and write QPS and RT for that view.

Real-time feature link

The storage service provided by FeatureStore mainly includes three parts: Feature Service (access layer), MSMQ (DataHub), and FeatureDB.

In real-time features, users can call the feature service through FeatureStore Java SDK or Flink Connector to write feature data to FeatureDB. Data written through feature service will also be synchronized to the user's MaxCompute table, which can be used for real-time feature sample export and further model training.

For feature data stored in FeatureDB, users can read it through FeatureStore's Java/Go SDK, or pull all features through EasyRec Processor and store them in local cache for higher performance reading. For real-time features, the latest feature information can be obtained at the millisecond level.

Real-time feature lifecycle

When creating a real-time feature view, you can specify Feature Lifecycle for the FeatureDB table. When the survival time of a row reaches the lifecycle, it will be automatically cleaned up within seconds.

You can specify the survival time using the following methods:

• Method 1: Do not set an Event Time field. In this case, the survival time will be calculated based on the data's write time.

• Method 2: Check Event Time for a feature field. The unit is milliseconds. Assuming event_time is the value of Event Time, time_now is the current time, and time_ttl = time_now - ttl is the event_time when the data should start to expire. The specific handling methods for written feature data are:

  • If using PartialFieldWrite mode for partial field update writing, the survival time will be based on the actual data write time.

  • event_time > time_now + 15min: Data will not be written. (This prevents timestamp differences between different systems, allowing a 15-minute buffer)

  • time_ttl < event_time <= time_now + 15min: Data is written normally, survival time is calculated starting from event_time, and the row of data is automatically cleaned up after reaching the lifecycle.

  • 0 < event_time < time_ttl: Data will be automatically cleaned up after being written. Note that the unit of event_time is milliseconds. If the value of your Event Time field is in seconds, it will fall into this case, causing the data to not be written successfully.

  • event_time <= 0: Survival time is calculated based on the actual data write time.

  • Invalid value (cannot be converted to integer): Data will not be written.

  • Registered an Event Time field but did not pass in the value of the Event Time field: Data is written normally, survival time is calculated based on the actual data write time.

  • No Event Time field: Data is written normally, survival time is calculated based on the actual data write time.

  • Additionally, in FeatureDB, the value of event_time is used the ts for this row of data, which means that if you need to update data corresponding to a key, the value of the Event Time field needs to be equal or greater than the previous value for this row of data to be updated. If the new event_time < the original event_time value, the data corresponding to this key will not be updated.