plugins

Auto.js provides a mechanism for loading plug-ins, allowing users to write apk with Activity, Service, C/C++ library, etc., install it on an Android device, and load and call it with Auto.js.

A plug-in is an apk file that can be installed independently. After the user installs it, the plug-in is loaded and the API is called through the $plugins module.

$plugins.load(packageName)

• packageName {string} Loaded plugin package name

Load a plug-in and return the object exported by module.exports in the plug-in module.

If the plug-in is not installed, a PluginLoadException exception will be thrown.

The following is an example of the use of OCR plug-in: (see [OCR plug-in](#ocr plug-in) for the plug-in download address)

requestScreenCapture();

let OCR = $plugins.load('org.autojs.plugin.ocr');
let ocr = new OCR();
// Capture the screen and recognize the text on the screen, and return the recognition result
console.log(ocr.ocrScreen());

How to develop a plugin

The following sample code can be found here for the complete project: Plugin SDK

The package names in this example are all org.autojs.plugin.sdk.demo, the plug-in package name may be different in the actual project.

Plug-in SDK integration

Create a new Android project and add in the project's build.gradle file:

allprojects {
    repositories {
        // ...
        maven {url'https://jitpack.io'}
    }
}

Add in the build.gradle file of the specific module (such as app):

dependencies {
    // ...
    implementation'com.github.hyb1996:Auto.js-Plugin-SDK:0.2'
}

For more information, see Jitpack.io.

Plug-in configuration

1. Create a new PluginHelloWorld file, inherited from Plugin.

public class PluginHelloWorld extends Plugin {

    public PluginHelloWorld(Context context, Context selfContext, Object runtime, Object topLevelScope) {
        super(context, selfContext, runtime, topLevelScope);
    }

    // Return the assets directory path of the JavaScript glue layer code of the plugin
    @Override
    public String getAssetsScriptDir() {
        return "plugin-helloworld";
    }

    // Plug-in public API, which can be called by JavaScript code
    public String getStringFromJava() {
        return "Hello, Auto.js!";
    }

    // Plug-in public API, which can be called by JavaScript code
    public void say(String message) {
        getSelfContext().startActivity(new Intent(getSelfContext(), HelloWorldActivity.class)
                .putExtra("message", message)
                .addFlags(Intent.FLAG_ACTIVITY_NEW_TASK));
    }
}

2. Added MyPluginRegistry file, inherited from PluginRegistry:

public class MyPluginRegistry extends PluginRegistry {

    static {
        // Register the default plugin
        registerDefaultPlugin(new PluginLoader() {
            @Override
            public Plugin load(Context context, Context selfContext, Object runtime, Object topLevelScope) {
                return new PluginHelloWorld(context, selfContext, runtime, topLevelScope);
            }
        });
    }
}

Configure the following meta-data in AndroidManifest.xml, name is "org.autojs.plugin.sdk.registry", and value is the package name of MyPluginRegistry.

  <application
        ...>

        <meta-data
            android:name="org.autojs.plugin.sdk.registry"
            android:value="org.autojs.plugin.sdk.demo.MyPluginRegistry" />

        <activity
            android:name=".HelloWorldActivity"
            android:exported="true" />

        <service
            android:name=".HelloworldPluginService"
            android:exported="true" />
    </application>

3. Write JavaScript glue layer

Add the index.js file in the corresponding directory of assets (returned by Plugin.getAssetsScriptDir), which is used as a glue layer to export the plugin API.

module.exports = function (plugin) {
    let runtime = plugin.runtime;
    let scope = plugin.topLevelScope;

    function helloworld() {
    }

    helloworld.stringFromJava = plugin.getStringFromJava();

    helloworld.say = function (message) {
        plugin.say(message);
    }

    return helloworld;
}

4. Call in Auto.js Pro

Compile the plug-in as apk (assembleDebug/assembleRelease) and install it on the device. Use the following code call in Auto.js Pro:

let helloworld = $plugins.load("org.autojs.plugin.sdk.demo");
console.log(helloworld.stringFromJava);
helloworld.say("Hello, Auto.js Pro Plugin");

5. Independent service AIDL method call

You can write a Service in the plug-in, which is invoked and bound by Auto.js Pro, and the Service interface can be called through aidl in js.

Override the method getService in Plugin.

// Plug-in service class, optional, used to communicate with Auto.js Pro ontology in AIDL mode. Can return null
@Override
public ComponentName getService() {
    return new ComponentName(getSelfContext().getPackageName(), HelloworldPluginService.class.getName());
}

Create a new Service component (Note that it needs to be exported="true" when registering in AndroidManifest, inherited from PluginService.

public class HelloworldPluginService extends PluginService {
    private static final String ACTION_ADD = "add";

    @Override
    protected Result onRemoteCall(@NonNull String action, @NonNull Map<String, Object> args, @Nullable RemoteCallback callback) throws RuntimeException {
        switch (action) {
            case ACTION_ADD:
                return invokeAdd(args);
        }
        return Result.notImplemented(action);
    }

    private Result invokeAdd(Map<String, Object> args) {
        Number a = PluginUtils.getNotNull(args, "a");
        Number b = PluginUtils.getNotNull(args, "b");
        double sum = a.doubleValue() + b.doubleValue();
        return new Result(Collections.singletonMap("sum", sum));
    }
}

Add glue layer code in index.js:

helloworld.remoteAdd = function (a, b) {
    return plugin.waitForConnection().call('add', {
        a: a,
        b: b
    }, null).get('sum');
}

Then you can call in Auto.js Pro:

let helloworld = $plugins.load("org.autojs.plugin.sdk.demo");
console.log(helloworld.remoteAdd(1, 2));

OCR plugin

At present, the Auto.js official only provides a simple OCR plug-in as an example, its efficiency is relatively slow, only for reference.

OCR plug-in download(Password: 4fw3)

new OCR([dataFile, lang])

• dataFile {string} data path
• lang {string} language

Create a new OCR object. The OCR plugin uses Tesseract as the engine. The dataFile and lang parameters of the constructor are used to initialize the Tesseract engine with the specified language model.

The data path dataFile must be the path of the parent directory of tessdata and must end with /. Any path after the last/will be ignored.

To load multiple languages, for example, hin + eng will load Hindi and English.

For more information, see TessBaseAPI.

Use the built-in Chinese character library data to initialize OCR:

let OCR = $plugins.load('org.autojs.plugin.ocr');
let ocr = new OCR();

Initialize OCR with your own data:

let OCR = $plugins.load('org.autojs.plugin.ocr');
let ocr = new OCR("/sdcard/tessdata/", "chi_sim");

Pytorch plugin

This plugin and documentation are developed and provided by the third-party developer Haoran, thank you very much.

Pytorch plug-in download(Password: 4n6x)

To use this module, please install the Pytorch-AjPlugin extension plug-in. And use let pytorch = $plugins.load("com.hraps.pytorch") code to load.

The Pytorch module provides the functions that the completed deep learning neural network model executes on Android devices, and can realize functions that are difficult to achieve by conventional programs, such as: image recognition, language translation, language question and answer, etc.

Before use, you need to make sure that you have already trained the neural network model. Use Python to convert the model file into an Android script file, and use the Python function torch.jit.trace(model, input_tensor).

If you don’t know much about Pytorch, I (referring to the plugin author Haoran) recommend you to go to this website to learn about: Deep Learning Pytorch

You can test the device supportability through pytorch.debugOd(), and you can also see the high-quality recognition effect of this module on image detection.

Pytorch weight export guide

The .pt file needs to be converted to the .torchscript.pt file to be adapted on the mobile terminal. Through tensor flow tracking, the model can be converted to the mobile terminal for use. The model is not friendly to third-party support libraries. It is recommended to use pure Pytorch to write neural network models and train them.

The following is the computer-side Python conversion script.

model = make_model('resnet18') # Import model structure
model.load_state_dict(torch.load(model_pt)) # Load training weight parameters
model.eval() # The model is set to evaluation mode, which must be turned on
input_tensor = torch.rand(1,3,224,224) # Set the input data format, here the 224*224 rgb3 channel image format is simulated to generate a random tensor.
mobile = torch.jit.trace(model, input_tensor,strict=False) # Model conversion
mobile = optimize_for_mobile(mobile) # mobile terminal optimization (optional)
mobile.save(mobile_pt) # save the file

Basic functions

pytorch.load(path[,device])

• path {String} The path where the model is located
• device {int} Execution device, 0 is CPU, 1 is VULKAN, default is 0.
• Return {PytorchModule}

Import the network model. Only a few devices support VULKAN, and the device parameter may not be used.

When the value of path is "yolov5s", the built-in target detection model will be imported, and when the value of path is "textcnn, the built-in sentiment analysis model will be imported."

pytoch.forward(module,input)

• module {PytorchModule} The imported neural network model.
• input {Tensor} Model input tensor.
• Returns the output tensor of the {Tensor} model.

Execute the network model to move forward and get the calculation result.

pytoch.forwardTuple(module,input)

• module {PytorchModule} The imported neural network model.
• input {Tensor} Model input tensor.
• Returns the output tensor of the {Tensor} model.

Execute the network model to move forward and get the calculation result.

The difference with pytorch.forward() is that it returns the first item of the tuple in the output, which is suitable for the corresponding model, such as the target detection model. His essence is actually module.forward(IValue.from(inputTensor)).toTuple()[0].toTensor().

pytorch.destory(module)

• module {PytorchModule} The neural network model to be released.

Unleash the neural network.

Tensor class

The tensor class is the general input and output data structure of the neural network, which is convenient for the high-speed processing of the network, and is a high-dimensional array. For example, if a picture is 100200 in size and has 3 RGB channels, it will be converted into a 100200*3 long floating-point number array before being passed into the neural network.

pytorch.fromBlob(arr,shape)

• arr {List} js array
• shape {List<Long>} The shape of the transformed tensor
• Return the tensor generated by {Tensor}

Construct tensor from js array.

tensor.getDataAsFloatArray()

• Return {List<float>} return the float array converted from tensor

Convert the tensor to a floating-point array.

tensor.getDataAs[Byte/Double/Float/Int/Long]Array()

• Return {List<...>}\ returns the array after tensor conversion

Convert tensors into arrays of various types.

ObjectDetection Object Detection-Function Function

Target detection is used to analyze the position and type of various items in the image. If you don’t understand, you can use pytorch.debugOd() to check the effect.

pytorch.debugOd([modulePath,classPath])

• modulePath {String} Model file path (.pt/.pth file)
• classPath {String} tag name file path (.txt file)

Test the target detection model file and start the built-in debugging page.

Empty incoming can use built-in target detection weights to test equipment support. That is, directly pytorch.debugOd().

pytorch.liveOd(modulePath,classPath)

• modulePath {String} Model file path (.pt/.pth file)
• classPath {String} tag name file path (.txt file)

Enter the camera real-time recognition page to view the dynamic push results.

ObjectDetection(OD) Object Detection-Common Functions

It is used to build a target detection network by itself and provide commonly used functions.

pytorch.getIOU(a,b)

• a {Rect} range a
• b {Rect} range b
• Return {float} IOU value

Calculate the overlap ratio (IOU value) of the two rectangles, and the overlap area is more than the total area.

pytorch.bitmapToTensor(img[,mean,std])

• img {Bitmap} original image
• mean {List<float>} normalized average value, the default value is [0.485, 0.456, 0.406]
• std {List<float>} Normalized standard deviation, the default value is [0.229, 0.224, 0.225]
• Return {Tensor} Tensor tensor after image conversion

Convert the image into a Tensor tensor type for easy input to the network model. The image needs to use the bitmap type, such as the image type in Autojs, which can be converted to bitmap through image.getBitmap().

The mean and std values ​​are used to normalize the color of the picture to a certain range. Please set them uniformly according to the model training value. It is not clear that it can be set to [0,0,0] [1,1,1].

img = images.read("/sdcard/a.jpg");
inputTensor = pytorch.bitmapToTensor(img.getBitmap(),[0,0,0],[1,1,1]);

pytorch.floatsToResults(floats,row,column,imgScaleX,imgScaleY[,threshold])

• floats {List<float>} Yolo model output array
• row {int} the number of output results
• column {int} the amount of data for each result
• imgScaleX {float} Image X-axis scaling
• imgScaleY {float} Y-axis scaling of the image
• threshold {float} keep the lowest confidence value of the result
• Return all output results of {List<OdResult>}

Convert the output result array of the Yolo model into a recognition result class. Floats is the total output data, and floats.length should be equal to row*column.

When inputting the model, the picture is compressed into a fixed input size of the network. Here, the coordinates of the result position in the original image can be calculated inversely by inputting the XY value of imgScale.

The output of the Yolo model is composed of detection blocks, each detection block will fix the prediction result of the output result center point in it, and each detection result is composed of confidence, position, and probability of each type. For example, in the built-in Yolo model, each output contains the frame (x, y, w, h) of the result, a total of 4 values, and the confidence (that is, the correct rate) of the result is 1 value, and the result is for each type Probability, if the output will have data such as [car:0.2,bus:0.9,boat:0.1], the built-in model can identify a total of 80 items. So the final column value is 4+1+80=85. The row block is the number of detection blocks, and each detection block has column output numbers.

pytorch.useNMS(results[,limit,threshold])

• results {List<OdResult>} all output results
• limit {int} Maximum remaining result limit
• threshold {float} Border repeat rate threshold
• Return {List<OdResult>} NMS processed result

Filter the results with probabilities of repetition. The NMS algorithm is NonMaxSuppression, which removes the result of the repetition rate higher than the threshold in the frame.

OdResult class

A class used to represent a single result of target detection, a compound class containing three parameters: rect, score, and classId.

odResuult.score

• score {float} Get the confidence of the result, similar to the correct rate

odResult.rect

• rect {Rect} Get the frame position of the result, the document about the rect can be seen in the images module

odResult.classIndex

• classIndex {int} The type number of the target

NaturalLanguageProcessing(NLP) Natural language processing function

Provides related functions of natural language processing models

pytorch.debugTec([modulePath,vocabPath])

• modulePath {String} Model file path (.pt/.pth file)
• vocabPath {String} Vocabulary serial number file path (.txt file)

Test the natural language sentiment analysis TextCNN model file and start the built-in debugging page.

Empty incoming can use built-in sentiment analysis weights to test device support. That is, directly pytorch.debugTEC().

pytorch.simplifySentence(sentence)

• sentence {String} Input sentence.
• Return {String} the simplified sentence.

Simplify English sentences. Only keep alphanumeric characters and reduce them to lowercase.

Vocab class

The vocabulary class provides a more efficient conversion function between vocabulary and serial number.

pytorch.vocabPath(path)

• path {String} Glossary file path
• Return {Vocab} vocabulary entity

The file should be composed of one word per line, and the correspondence between line numbers and words should be consistent with the word vector file used during model training.

pytorch.vocab(words)

• words {List<String>} vocabulary list
• Return {Vocab} vocabulary entity

vocab.size()

• Return {long} vocabulary size

Get the amount of vocabulary contained in the vocabulary.

vocab.getWord(id)

• id {long} vocabulary number
• Return {String} vocabulary text

Get the vocabulary corresponding to the vocabulary number in the vocabulary list.

vocab.getId(word)

• word {String} vocabulary text
• Return {long} vocabulary number

Get the serial number corresponding to the vocabulary text in the vocabulary.

vocab.getWords(ids[,length])

• ids {List<long>} vocabulary number
• length {int} returns the length of the list
• Return {List<String>} vocabulary text

Get a vocabulary text list of multiple vocabulary numbers in the vocabulary.

vocab.getIds(words[,length])

• words {List<String>} vocabulary text
• length {int} Return the length of the list, fill in the vacancy with 0.
• Return {List<long>} vocabulary number

Get a list of vocabulary serial numbers of multiple vocabularies in the vocabulary list.

Usage examples

Image target detection (Yolov5s model)

/**
 * Pytorch plug-in target detection algorithm Yolov5 model implementation example
 *
 * Author: Hao Ran (Q: 2125764918)
 */



//To experience the visual recognition effect of this plug-in, you can use these three lines of code:
/*
pytorch = $plugins.load("com.hraps.pytorch")
pytorch.debugOd()
exit()
*/


//Import the plug-in module
pytorch = $plugins.load("com.hraps.pytorch")

//Import neural network model input model file path (here imported built-in Yolov5s model)
var model = pytorch.load("yolov5s")
//Import the type name corresponding to the recognition result (a string array composed of the class name, you can write it by yourself, such as ["car","plane","person"...])
var classes = pytorch.getCocoClasses()

//Define the side length of the input image of the model. The input dimension is w*h*3
var inputWidth = 640
var inputHeight = 640
//Define the number of model outputs and the size of each output dimension is row*column
//row is the number of divisions of the yolo model, which is related to the input size
var outputRow = 25200
//column is the dimension of each grid, consisting of 4 values ​​for position (x, y, w, h), 1 value for score (score), 80 values ​​for type (coco data set 80 classes), a total of 85 value
var outputColumn = 4 + 1 + 80

//Import the picture to be recognized
var img = images.read("/sdcard/DCIM/Camera/b.jpg")
//Zoom to model input dimension
var inputImg = images.resize(img, [inputWidth, inputHeight])
//The image is converted to a tensor. The MEAN and STD values ​​are set to 000 and 111, that is, special normalization is not enabled
inputTensor = pytorch.bitmapToTensor(inputImg.getBitmap(), [0, 0, 0], [1, 1, 1])
//Perform neural network advancement to obtain output tensor
output = pytorch.forwardTuple(model, inputTensor)
//Tensor to float array
f = output.getDataAsFloatArray()
log("Model output dimension: "+ f.length)

//Calculate the graphics zoom ratio
imgScaleX = img.getWidth() / inputWidth
imgScaleY = img.getHeight() / inputHeight
//Restore the real position of the recognition result and convert it to the target detection result class array
results = pytorch.floatsToResults(f, outputRow, outputColumn, imgScaleX, imgScaleY)
log("The number of initial network recognition: "+ results.size())
//NMS algorithm filters duplicate results
nmsResults = pytorch.useNMS(results)
toastLog("Number of final results: "+ nmsResults.size())
//Traverse the output result
for (var i = 0; i <nmsResults.size(); i++) {
    result = nmsResults.get(i)
    rect = result.rect
    str = "Type: "+ classes.get(result.classIndex) +" Confidence: "+ result.score +" Position: left" + rect.left + "up" + rect.top + "right" + rect. right + "下" + rect.bottom;
    log(str)
}

Natural language sentiment analysis (TextCNN model)

/**
 * Pytorch plug-in Language sentiment analysis algorithm TextCNN model implementation example
 *
 * Author: Hao Ran (Q: 2125764918)
 */


//To experience the effects of this plug-in, you can use this line of code:
/*
pytorch = $plugins.load("com.hraps.pytorch")
pytorch.debugTec()
exit()
*/


//Sentence to be recognized
var text = "The program is useful!"

//Import the plug-in module
pytorch = $plugins.load("com.hraps.pytorch")
//Import neural network model input model file path (here, the built-in textcnn model is imported)
var model = pytorch.load("textcnn")
//Imported vocabulary is composed of a txt file with one line and one word. The line number and word should correspond to the sequence number of the word vector during model training (the vocabulary list corresponding to the built-in model is imported here)
var vocab = pytorch.getTextcnnVocab()
log("The vocabulary has been imported successfully, total" + vocab.size() + "pieces");

//Simplified sentence processing, filter punctuation, all lowercase
var textSimple = pytorch.simplifySentence(text)
log("Simplified sentence: "+ textSimple);
//Define the length of the model input vocabulary
var inputSize = 128
//According to the vocabulary, replace the sentence with a word vector sequence number. Unknown words and insufficient lengths are automatically filled with 0
var ids = vocab.getIds(textSimple.split(" "), inputSize)
log(ids)
//Construct the list into Tensor type, suitable for network input
var inputTensor = pytorch.fromBlob(ids, [1, 128])
//Execute the model to get the output result
var outPutTensor = pytorch.forward(model, inputTensor)
//The output tensor is converted to a floating-point array
var result = outPutTensor.getDataAsFloatArray()
log("Model output: "+ result[0] +" "+ result[1])
//Analysis results
console.info("Statement:" + text)
if (result[0] <= result[1]) {
    console.info("The result is: positive emotion")
} else {
    console.info("The result is: negative emotions")
}