LLM - Copyright Removal (DLC) component - Platform For AI

The LLM - Copyright Removal (DLC) component removes copyright information from text. It is typically used to strip copyright comment blocks from the beginning of code files. Input data from OSS must be in JSON Lines (JSONL) format (Example), where each line is a valid JSON object.

Supported compute resources

DLC

How it works

The component removes copyright information and other leading comments from text in two steps:

First, the component scans the text for any strings that match the regular expression '/\\*[^*]*\\*+(?:[^/*][^*]*\\*+)*/', which identifies block comments.
- If a matching string is found, the component checks if it contains the word copyright. If it does, the entire comment block is removed. Otherwise, the string is left unchanged.
- If no match is found, the component proceeds to step 2.
The component splits the text into individual lines, searches for the first block of consecutive comment lines (lines starting with //, #, or --), and removes the entire block.

Both steps only process the first comment block found, which is assumed to be at the beginning of the text. The rest of the content is left unchanged. For example:

Before processing

/**
 * angular-spinner version 0.3.1
 * License: MIT.
 * Copyright (C) 2013, 2014, Uri Shaked and contributors.
 */

(function(window, angular, undefined) {
'use strict';

angular.module('angularSpinner', [])

After processing

(function(window, angular, undefined) {
'use strict';
angular.module('angularSpinner', [])
.factory('usSpinnerService', ['$rootScope', function ($rootScope) {
  var config = {};
  config.spin = function (key) {
    $rootScope.$broadcast('us-spinner:spin', key);
  };
}]);
})(window, angular);

Configure the component

In Designer, add the LLM - Copyright Removal (DLC) component to your workflow and configure its parameters in the right-hand panel.

Parameter group	Parameter		Required	Description	Default
Field settings	Target field		Yes	The name of the field to be processed.	N/A
Field settings	Output OSS directory		No	The OSS directory for storing the processed data. If left empty, the default workspace path is used.	N/A
Execution tuning	Number of processes		No	The number of parallel processes to use for the job.	8
	Select resource group	Public resource group	No	Select the node specification (CPU or GPU instance type), number of nodes, and VPC.	N/A
	Select resource group	Dedicated resource group	No	Select the number of CPU cores, memory, shared memory, number of GPUs, and number of nodes.	N/A
	Maximum runtime		No	The maximum runtime for the job. The system terminates the job if it exceeds this duration.	N/A