images - Image processing

Stability: 2 - Stable

The images module provides common image-processing utilities for mobile devices, including screenshots, reading/writing images, cropping, rotation, thresholding/binarization, and color/image search.

This module is roughly divided into two parts: (1) image/color searching and (2) image processing.

Note: after creating an Image object, recycle it as soon as you no longer need it, and avoid creating lots of images in a loop. Images consume a lot of memory. Although Auto.js tries to reduce leaks and memory usage (e.g. image caching, recycling during GC, and recycling all images when the script ends), poorly written code can still consume large amounts of memory.

Recycle an Image by calling recycle(), for example:

// Read image
var img = images.read("./1.png");
// Process image
...
// Recycle image
img.recycle();

Exception: images returned by captureScreen() do not need to be recycled.

Image processing

images.read(path)

• path {string} Image file path

Reads an image file at path and returns an Image object. Returns null if the file does not exist or cannot be decoded.

images.load(url)

• url {string} Image URL

Loads an image from URL and returns an Image object. Returns null if the URL does not exist or the image cannot be decoded.

images.copy(img)

• img {Image} Image
• Returns {Image}

Creates a full copy of an image and returns the new Image. This function copies all pixel data from img.

images.save(image, path[, format = "png", quality = 100])

• image {Image} Image

• path {string} Output path

• format {string} Image format. Supported values:

  • png
  • jpeg/jpg
  • webp

• quality {number} Image quality, an integer in the range 0~100

Saves image to path. The file will be created if it does not exist, and overwritten if it already exists.

// Compress to half quality and save
var img = images.read("/sdcard/1.png");
images.save(img, "/sdcard/1.jpg", "jpg", 50);
app.viewFile("/sdcard/1.jpg");

images.fromBase64(base64)

• base64 {string} Base64-encoded image data
• Returns {Image}

Decodes Base64 data and returns an Image. Returns null if the data cannot be decoded.

images.toBase64(img[, format = "png", quality = 100])

• image {image} Image
• format {string} Image format. Supported values:
  • png
  • jpeg/jpg
  • webp
• quality {number} Image quality, an integer in the range 0~100
• Returns {string}

Encodes an image to Base64 and returns the string.

images.fromBytes(bytes)

• bytes {byte[]} Byte array

Decodes a byte array and returns an Image. Returns null if the data cannot be decoded.

images.toBytes(img[, format = "png", quality = 100])

• image {image} Image

• format {string} Image format. Supported values:

  • png
  • jpeg/jpg
  • webp

• quality {number} Image quality, an integer in the range 0~100

• Returns {byte[]}

Encodes an image to a byte array and returns it.

images.readPixels(path)

• path {string} Image path
• Returns {Object} Includes pixel data and width/height: {data,width,height}

Reads pixel data and the image size (width/height).

images.clip(img, x, y, w, h)

• img {Image} Image
• x {number} X of top-left corner of the crop region
• y {number} Y of top-left corner of the crop region
• w {number} Width of the crop region
• h {number} Height of the crop region
• Returns {Image}

Crops a region of size w * h from img at position (x, y), and returns the cropped image.

var src = images.read("/sdcard/1.png");
var clip = images.clip(src, 100, 100, 400, 400);
images.save(clip, "/sdcard/clip.png");

images.resize(img, size[, interpolation])

[v4.1.0 Added]

• img {Image} Image
• size {Array} Two-element array [w, h] for width/height. If it has only one element, width and height are the same.
• interpolation {string} Interpolation method (optional). Default is "LINEAR". Supported values:
  • NEAREST Nearest-neighbor interpolation
  • LINEAR Linear interpolation (default)
  • AREA Area-based interpolation
  • CUBIC Cubic interpolation
  • LANCZOS4 Lanczos interpolation. See InterpolationFlags
• Returns {Image}

Resizes an image and returns the resized image. For example, to resize to 200*300: images.resize(img, [200, 300]).

See Imgproc.resize.

images.scale(img, fx, fy[, interpolation])

[v4.1.0 Added]

• img {Image} Image
• fx {number} Scale factor in X direction
• fy {number} Scale factor in Y direction
• interpolation {string} Interpolation method (optional). Default is "LINEAR". Supported values:
  • NEAREST Nearest-neighbor interpolation
  • LINEAR Linear interpolation (default)
  • AREA Area-based interpolation
  • CUBIC Cubic interpolation
  • LANCZOS4 Lanczos interpolation. See InterpolationFlags
• Returns {Image}

Scales an image and returns the scaled image. For example, to scale to half: images.scale(img, 0.5, 0.5).

See Imgproc.resize.

images.rotate(img, degree[, x, y])

[v4.1.0 Added]

• img {Image} Image
• degree {number} Rotation angle (degrees)
• x {number} Rotation center X. Default is image center.
• y {number} Rotation center Y. Default is image center.
• Returns {Image}

Rotates the image counter-clockwise by degree degrees and returns the rotated image.

For example, to rotate 90 degrees counter-clockwise: images.rotate(img, 90).

images.concat(img1, image2[, direction])

[v4.1.0 Added]

• img1 {Image} Image 1
• img2 {Image} Image 2
• direction {string} Concatenation direction. Default is "RIGHT". Supported values:
  • LEFT Attach image 2 to the left of image 1
  • RIGHT Attach image 2 to the right of image 1
  • TOP Attach image 2 above image 1
  • BOTTOM Attach image 2 below image 1
• Returns {Image}

Concatenates two images and returns the result. If the two images have different sizes, the smaller one will be centered appropriately.

images.grayscale(img)

[v4.1.0 Added]

• img {Image} Image
• Returns {Image}

Converts an image to grayscale and returns the result.

images.threshold(img, threshold, maxVal[, type])

[v4.1.0 Added]

• img {Image} Image
• threshold {number} Threshold
• maxVal {number} Max value
• type {string} Threshold type. Default is "BINARY". See ThresholdTypes. Supported values:
  • BINARY
  • BINARY_INV
  • TRUNC
  • TOZERO
  • TOZERO_INV
  • OTSU
  • TRIANGLE
• Returns {Image}

Applies thresholding and returns the processed image. You can use it for binarization. For example: images.threshold(img, 100, 255, "BINARY") turns values greater than 100 into 255 and all others into 0. If img is already grayscale, the result will be a black-and-white image.

See also: threshold usage (blog) or OpenCV docs: threshold.

images.adaptiveThreshold(img, maxValue, adaptiveMethod, thresholdType, blockSize, C)

[v4.1.0 Added]

• img {Image} Image
• maxValue {number} Max value
• adaptiveMethod {string} Algorithm used to compute the threshold within a neighborhood. Supported values:
  • MEAN_C Mean of neighborhood minus C
  • GAUSSIAN_C Gaussian-weighted mean of neighborhood minus C
• thresholdType {string} Threshold type. Supported values:
  • BINARY
  • BINARY_INV
• blockSize {number} Neighborhood block size
• C {number} Offset adjustment
• Returns {Image}

Applies adaptive thresholding and returns the processed image.

See also: threshold vs adaptiveThreshold (blog) or OpenCV docs: adaptiveThreshold.

images.cvtColor(img, code[, dstCn])

[v4.1.0 Added]

• img {Image} Image

• code {string} Color space conversion code. There are 205 options in total (see ColorConversionCodes); here are a few examples:

  • BGR2GRAY Convert BGR to grayscale
  • BGR2HSV Convert BGR to HSV

• dstCn {number} Number of channels in the destination image. If omitted, it is determined automatically.

• Returns {Image}

Converts the image color space and returns the converted image.

See also: color space conversion (blog) or OpenCV docs: cvtColor.

images.inRange(img, lowerBound, upperBound)

[v4.1.0 Added]

• img {Image} Image
• lowerBound {string} | {number} Lower bound color
• upperBound {string} | {number} Upper bound color
• Returns {Image}

Binarizes the image: colors outside lowerBound~upperBound become 0, and colors within the range become 255.

For example: images.inRange(img, "#000000", "#222222").

images.interval(img, color, interval)

[v4.1.0 Added]

• img {Image} Image
• color {string} | {number} Color value
• interval {number} Range interval for each channel
• Returns {Image}

Binarizes the image: colors outside color-interval ~ color+interval become 0, and colors within the range become 255. The plus/minus is applied per channel.

For example, images.interval(img, "#888888", 16): each channel is 0x88. After ±16, the range is [0x78, 0x98], so colors #787878~#989898 become #FFFFFF, and everything outside becomes #000000.

images.blur(img, size[, anchor, type])

[v4.1.0 Added]

• img {Image} Image
• size {Array} Kernel size, e.g. [3, 3]
• anchor {Array} Anchor point (smoothed point). Default is image center.
• type {string} Border extrapolation type. Default "DEFAULT". Supported values:
  • CONSTANT iiiiii|abcdefgh|iiiiiii with some specified i
  • REPLICATE aaaaaa|abcdefgh|hhhhhhh
  • REFLECT fedcba|abcdefgh|hgfedcb
  • WRAP cdefgh|abcdefgh|abcdefg
  • REFLECT_101 gfedcb|abcdefgh|gfedcba
  • TRANSPARENT uvwxyz|abcdefgh|ijklmno
  • REFLECT101 same as BORDER_REFLECT_101
  • DEFAULT same as BORDER_REFLECT_101
  • ISOLATED do not look outside of ROI
• Returns {Image}

Blurs (smooths) the image and returns the processed image.

See also: image smoothing (blog) or OpenCV docs: blur.

images.medianBlur(img, size)

[v4.1.0 Added]

• img {Image} Image
• size {Array} Kernel size, e.g. [3, 3]
• Returns {Image}

Applies median blur and returns the processed image.

See also: image smoothing (blog) or OpenCV docs: blur.

images.gaussianBlur(img, size[, sigmaX, sigmaY, type])

[v4.1.0 Added]

• img {Image} Image
• size {Array} Kernel size, e.g. [3, 3]
• sigmaX {number} Standard deviation in X direction. If omitted, it is computed automatically.
• sigmaY {number} Standard deviation in Y direction. If omitted, it is computed automatically.
• type {string} Border extrapolation type. Default is "DEFAULT". See images.blur.
• Returns {Image}

Applies Gaussian blur and returns the processed image.

See also: image smoothing (blog) or OpenCV docs: GaussianBlur.

images.getSimilarity(img1, img2, options)

• img1 {Image} Image 1
• img2 {Image} Image 2
• options {Object} Options:
  • type {string} Similarity algorithm (default: MSSIM):
    • MSSIM Mean structural similarity. SSIM is in the range ([0, 1]). The more similar the images, the closer it is to 1.
    • PNSR Peak signal-to-noise ratio (PSNR). Defined via mean squared error (MSE) for absolute pixel error. If PSNR < 30, the two images can be considered relatively similar.
• Returns {number}

Compares the similarity of two images and returns a similarity score.

Example:

log(
  images.getSimilarity(img1, img2, {
    type: "PNSR",
  }),
);

images.matToImage(mat)

[v4.1.0 Added]

• mat {Mat} OpenCV Mat object
• Returns {Image}

Converts an OpenCV Mat to an Image.

Image/Color search

This section introduces different approaches to locate targets using screenshots, matching, etc. Choose the one that fits your scenario:

• Find color / multi-point color search: match pixels by color or color-path description. High performance.
• Color search based on ColorMaping: best when you perform multiple color/multi-point searches per screenshot; highest efficiency.
• Find image (template matching): slide a small template over a large image to find the position. Medium efficiency, but weaker across different resolutions.
• Full-resolution image search (feature matching): match feature points between large and small images to compute position. Slower than template matching, but more robust to resolution/rotation changes.

images.requestScreenCapture([landscape])

• landscape {boolean} Whether the screenshot will be taken in landscape mode. false = portrait, true = landscape.

Requests screenshot permission from the system and returns whether the request succeeded.

On first use, the system permission prompt will appear. It is recommended to choose “Always allow” (some ROMs do not provide this option).

This function only requests permission; it does not take a screenshot. The actual screenshot API is captureScreen().

In a screenshot script, you only need to call this once. You do not need to call it every time before captureScreen(). If permission has already been granted, it will throw an exception.

If landscape is not specified, the screenshot orientation follows the current device orientation, so make sure the screen is in the desired orientation when you call this.

Screenshot permission cannot be shared across script engines.

It is recommended to run this function while your script UI/app is in the foreground. If you run it in another app's UI, a brief black screen may flash. Also, some custom ROMs or newer Android versions may not allow UI prompts in the background; calling this in the background may block indefinitely.

Example:

// Request screenshot permission
if (!requestScreenCapture()) {
  toast("Failed to request screenshot permission");
  exit();
}
// Capture 10 screenshots (1s interval) and save to storage
for (var i = 0; i < 10; i++) {
  captureScreen("/sdcard/screen_capture_" + i + ".png");
  sleep(1000);
}

This function can also be used as a global function.

$images.requestScreenCapture(options)

[Added in Pro 8.0]

• options {object} Screenshot request options

  • width {number} Screenshot width. Default -1 (auto = device screen width)

  • height {number} Screenshot height. Default -1 (auto = device screen height)

  • orientation {number} Screenshot orientation. Default 0

    • -1: ORIENTATION_CURRENT - detect and use current orientation
    • 0: ORIENTATION_AUTO - automatically adapt when orientation changes
    • 1: ORIENTATION_PORTRAIT - portrait
    • 2: ORIENTATION_LANDSCAPE - landscape

  • async {boolean} Whether to use async capture. Default false

Requests screenshot permission and returns whether it succeeded. For width and height, the system only chooses a nearby suitable size, so the actual size may not exactly match the specified values.

requestScreenCapture(\{orientation: 0\});

For more parameters and notes, see images.requestScreenCapture([landscape]) above. Here we only explain the async option.

When async is true, capture becomes asynchronous: you cannot use captureScreen(); instead, listen for the screen_capture event.

This event triggers automatically when the screen changes. It can save power for UIs that update infrequently; for games it may not provide the same benefit.

// Request permission, note async: true
requestScreenCapture(\{async: true\});
let target = $images.read('./test.png');
$events.on('exit', () => target.recycle());
// Listen for screen captures
$images.on("screen_capture", capture => {
    // Find image
    let pos = $images.findImage(capture, target);
    // Print
    console.log(pos);
});

$images.getScreenCaptureOptions()

[Added in Pro 8.8.12]

• Returns {object} | {null}

Gets current screenshot options. Returns null if screenshot permission has not been requested. The returned object includes: width, height, orientation (0: ORIENTATION_AUTO, 1: ORIENTATION_PORTRAIT, 2: ORIENTATION_LANDSCAPE), density, and async.

$images.stopScreenCapture()

[Added in Pro 8.8.12]

Releases screenshot permission. If permission was not requested, this does nothing.

images.captureScreen()

Captures the current screen and returns an Image.

If you call this without screenshot permission, it throws SecurityException.

This function never returns null. Two consecutive calls may return the same Image object because screen capture updates take time; calling repeatedly within a short interval (typically ~16ms) can yield the same frame.

Captures must be converted to Bitmap, so this call takes some time (0~20ms).

Also, after requestScreenCapture() succeeds, it takes some time before captures become available. If you call captureScreen() immediately, it may wait for a while (typically a few hundred ms) before returning.

Example:

// Request landscape screenshots
requestScreenCapture(true);
// Capture
var img = captureScreen();
// Get color at point (100, 100)
var color = images.pixel(img, 100, 100);
// Show the color
toast(colors.toString(color));

This function can also be used as a global function.

images.captureScreen(path)

• path {string} Screenshot output path

Captures the screen and saves it to path as PNG. The file will be created if it doesn't exist, and overwritten if it does.

This function returns nothing. It can also be used as a global function.

images.pixel(image, x, y)

• image {Image} Image
• x {number} Pixel x coordinate
• y {number} Pixel y coordinate

Returns the pixel ARGB value of image at (x, y).

The value format is 0xAARRGGBB (a "32-bit integer", though JavaScript does not distinguish integer/float types).

The coordinate origin is the top-left of the image. The left edge is the Y axis and the top edge is the X axis.

images.findColor(image, color, options)

• image {Image} Image
• color {number} | {string} RGB value to find. If it's an integer, it uses 0xRRGGBB (alpha channel is ignored). If it's a string, it uses "#RRGGBB".
• options {Object} Options

Finds color in the image. Returns the found Point, or null if not found.

Options:

• region {Array} Search region. An array with 2 or 4 elements. (region[0], region[1]) is the top-left corner; region[2] * region[3] is width/height. If region has only 2 elements, the region is from (region[0], region[1]) to the bottom-right of the image. If omitted, the whole image is searched.
• threshold {number} Color similarity threshold for matching, range 0~255 (smaller = more similar; 0 = exact match; 255 = any color matches). Default is 4. Convert to float similarity (0.0~1.0) via: similarity = (255 - threshold) / 255.

This function can also be used as a global function.

Example: loop until red is found.

requestScreenCapture();

// Loop and search for red (#ff0000)
while (true) {
  var img = captureScreen();
  var point = findColor(img, "#ff0000");
  if (point) {
    toast("Found red at (" + point.x + ", " + point.y + ")");
  }
}

Example: search within a region.

// Read local image /sdcard/1.png
var img = images.read("/sdcard/1.png");
// Check if image loaded successfully
if (!img) {
  toast("Image not found");
  exit();
}
// Find color within region: top-left (400, 500), size 300*200; threshold 4
var point = findColor(img, "#00ff00", {
  region: [400, 500, 300, 200],
  threshold: 4,
});
if (point) {
  toast("Found: " + point);
} else {
  toast("Not found");
}

images.findColorInRegion(img, color, x, y[, width, height, threshold])

A convenience wrapper for region-based color search.

Equivalent to:

images.findColor(img, color, {
  region: [x, y, width, height],
  threshold: threshold,
});

This function can also be used as a global function.

images.findAllPointsForColor(img, color, options)

• img {Image} Image
• color {number} | {string} Color to detect
• options {Object} Options:
  • region {Array} Search region. See images.findColor() for details.
  • similarity {number} Color similarity in range 0~1 (larger = more similar; 1 = exact match; 0 = any color matches).
  • threshold {number} Similarity threshold in range 0~255 (smaller = more similar). Default is 4. Convert via: similarity = (255 - threshold) / 255. Use either similarity or threshold; if both are present, similarity takes precedence.
• Returns {Array}

Finds all points whose color matches color. Returns an array of Points, or null if nothing is found.

Example: find all white points.

log(images.findAllPointsForColor(img, "#ffffff"));

images.findColorEquals(img, color[, x, y, width, height])

• img {Image} Image
• color {number} | {string} Color to find
• x {number} Region top-left X
• y {number} Region top-left Y
• width {number} Region width
• height {number} Region height
• Returns {Point}

Finds a point whose color equals color exactly within the specified region, and returns the point. Returns null if not found.

The region is defined by x, y, width, height. If omitted, it searches the whole image.

This function can also be used as a global function.

Example: determine whether QQ has unread messages by checking the red dot color.

requestScreenCapture();
launchApp("QQ");
sleep(1200);
var p = findColorEquals(captureScreen(), "#f64d30");
if (p) {
  toast("Unread messages");
} else {
  toast("No unread messages");
}

images.findMultiColors(img, firstColor, colors[, options])

• img {Image} Target image
• firstColor {number} | {string} Color of the first point
• colors {Array} Remaining points relative to the first point. Each element is [x, y, color].
• options {Object} Options:
  • region {Array} Search region. See images.findColor() for details.
  • threshold {number} Similarity threshold in range 0~255 (smaller = more similar). Default is 4. Convert via: similarity = (255 - threshold) / 255.

Multi-point color search (similar to the “multi-point” feature in some automation tools). The process:

1. Find a location (x0, y0) where the pixel color matches firstColor.
2. For each [x, y, color] in colors, check whether the pixel at (x + x0, y + y0) matches color. If all match, return (x0, y0). Otherwise, keep searching for another firstColor position and repeat step 1.
3. Return null if no match is found in the whole image.

For example, for images.findMultiColors(img, "#123456", [[10, 20, "#ffffff"], [30, 40, "#000000"]]), suppose the pixel at (100, 200) is #123456. If the pixel at (110, 220) is #ffffff and at (130, 240) is #000000, then the function returns point (100, 200).

To restrict to a region, pass options.region, e.g.:

var p = images.findMultiColors(
  img,
  "#123456",
  [
    [10, 20, "#ffffff"],
    [30, 40, "#000000"],
  ],
  {
    region: [0, 960, 1080, 960],
  },
);

images.detectsColor(image, color, x, y[, threshold = 16, algorithm = "diff"])

• image {Image} Image
• color {number} | {string} Color to detect
• x {number} X coordinate to check
• y {number} Y coordinate to check
• threshold {number} Similarity threshold. Default 16. Range 0~255.
• algorithm {string} Color matching algorithm:
  • "equal": exact match (only matches when the pixel equals color)
  • "diff": absolute-difference match (matches when (|R-R0|+|G-G0|+|B-B0| < threshold))
  • "rgb": RGB Euclidean distance (matches when distance (\le) threshold)
  • "rgb+": weighted RGB distance (LAB Delta E)
  • "hs": HS distance (hue/saturation in HSV)

Returns whether image matches color at (x, y). Useful for checking whether a specific position is a certain color.

Example: detect whether a Weibo post is liked.

requestScreenCapture();
// Find the "like" widget
var like = id("ly_feed_like_icon").findOne();
// Get center coordinates
var x = like.bounds().centerX();
var y = like.bounds().centerY();
// Capture
var img = captureScreen();
// Check if the color at that point is orange-red
if (images.detectsColor(img, "#fed9a8", x, y)) {
  // Already liked; do nothing
} else {
  // Otherwise click like
  like.click();
}

images.detectsMultiColors(img, x, y, firstColor, colors, options)

• img {Image} Target image
• x {number} X of the first point
• y {number} Y of the first point
• firstColor {number} | {string} Color of the first point
• colors {Array} Remaining points relative to the first point. Each element is [x, y, color].
• options {Object} Options:
  • region {Array} Search region. See images.findColor() for details.
  • threshold {number} Similarity threshold in range 0~255 (smaller = more similar). Default is 4. Convert via: similarity = (255 - threshold) / 255.
• Returns boolean

Multi-point color verification: returns whether colors at multiple points match, starting from (x, y) in img.

See images.findMultiColors() for the multi-point definition.

log(
  images.detectsMultiColors(img, 100, 200, "#000000", [
    [3, 4, "#123456"],
    [8, 10, "#ff0000"],
  ]),
);

images.findImage(img, template[, options])

[v8.5.5 Added]

• img {Image} Large image
• template {Image} Small image (template)
• options {Object} Options

Finds an image (template matching). Searches for template inside img. Returns the found position Point, or null if not found.

Options:

• threshold {number} Image similarity in range 0~1 (float). Default 0.9.
• region {Array} Search region. See images.findColor() for region.
• level {number} Usually you don't need to change this. If omitted, it is adjusted automatically based on image size. The algorithm uses an image pyramid; level is the pyramid depth. Larger level may improve speed, but can also cause failures (over-downscaling) or wrong positions. Only use it if you understand it and want to tune performance.

This function can also be used as a global function.

The simplest example:

var img = images.read("/sdcard/large.png");
var templ = images.read("/sdcard/template.png");
var p = findImage(img, templ);
if (p) {
  toast("Found: " + p);
} else {
  toast("Not found");
}

A slightly more advanced example (search within region):

auto();
requestScreenCapture();
var wx = images.read("/sdcard/wechat_icon.png");
// Back to home screen
home();
// Capture and search
var p = findImage(captureScreen(), wx, {
  region: [0, 50],
  threshold: 0.8,
});
if (p) {
  toast("Found the WeChat icon on the home screen: " + p);
} else {
  toast("WeChat icon not found on the home screen");
}

images.findImageInRegion(img, template, x, y[, width, height, threshold])

A convenience wrapper for region-based image search. Equivalent to:

images.findImage(img, template, {
  region: [x, y, width, height],
  threshold: threshold,
});

This function can also be used as a global function.

images.matchTemplate(img, template, options)

[v4.1.0 Added]

• img {Image} Large image
• template {Image} Small image (template)
• options {Object} Options:
  • threshold {number} Image similarity in range 0~1 (float). Default 0.9.
  • region {Array} Search region. See images.findColor() for region.
  • max {number} Maximum number of matches to return. Default 5.
  • transparentMask {boolean} Whether to enable transparent-template matching. When enabled, template may have a transparent background. [Pro 8.0 added].
  • level {number} Usually you don't need to change this. See images.findImage() for details.
• Returns {MatchingResult}

Searches for a template in a large image and returns a MatchingResult. Useful when you want multiple match locations. Use max to limit the result count, and then you can sort or pick best matches from the results.

images.findCircles(gray, options)

• gray {Image} Grayscale image
• options {Object} Options:
  • region {Array} Search region. An array with 2 or 4 elements. (region[0], region[1]) is the top-left corner; region[2] * region[3] is width/height. If it has only 2 elements, the region is from (region[0], region[1]) to the bottom-right. If omitted, the whole image is searched.
  • dp {number} Inverse ratio between accumulator resolution and original image resolution. dp=2 means accumulator resolution is half of the original in both width and height; dp=1 means the same. Default 1.
  • minDst {number} Minimum distance between circle centers. Default is one eighth of the image height.
  • param1 {number} Higher threshold for Canny edge detector; the lower one is set to half of it. Default 100, range 0-255.
  • param2 {number} Accumulator threshold for circle detection. Default 100.
  • minRadius {number} Minimum circle radius. Default 0.
  • maxRadius {number} Maximum circle radius. 0 means no limit. Default 0.
• Returns {Array}

Finds circles in an image (Hough circle transform). Returns an array of circles {x, y, radius}, or null if not found.

Example:

// Request screenshot permission
requestScreenCapture();
// Capture
let img = captureScreen();
// Convert to grayscale
let gray = images.grayscale(img);
// Find circles
let arr = findCircles(gray, {
  dp: 1,
  minDst: 80,
  param1: 100,
  param2: 100,
  minRadius: 50,
  maxRadius: 80,
});
// Recycle image
gray.recycle();

$images.detectAndComputeFeatures(img[, options]）

[v9.2 Added]

• img {Image} Image to compute features from
• options {object} Feature options (optional):
  • scale {number} Scale factor used during feature computation. Smaller = faster but may reduce accuracy due to over-scaling. For images where width * height > 1,000,000, default is 0.5; otherwise default is 1.
  • grayscale {boolean} Whether to convert to grayscale before computing features. Default true.
  • method {string} Feature method. Default SIFT. You may also set ORB (not recommended).
  • region {Array} Region to compute features for. If omitted, computes for the whole image.
• Returns {ImageFeatures} Feature object. Call recycle() when done.

Computes image features and returns the feature object. It can be used later with matchFeatures for feature matching.

In general, you should compute features for the small/object image at the start of the program and recycle them at the end. If you load the template and compute its features on every screenshot, it not only slows the script down, but also increases memory fragmentation, making memory harder to reuse and faster to exhaust.

$images.matchFeatures(scene, object[, options])

[v9.2 Added]

• scene {ImageFeatures} Scene feature object (large image)
• object {ImageFeatures} Object feature object (small image)
• options {object} Options (optional):
  • matcher {string} Matcher type. Default FLANNBASED. Supported values: "FLANNBASED", "BRUTEFORCE", "BRUTEFORCE_L1", "BRUTEFORCE_HAMMING", "BRUTEFORCE_HAMMINGLUT", "BRUTEFORCE_SL2". Matchers other than FLANNBASED are not fully tested.
  • drawMatches {string} Output path to draw match details. If empty, no drawing. Intended for debugging; do not enable in production matching because it increases runtime.
  • threshold {number} Match threshold. Default 0.7.
• Returns {ObjectFrame} | {null} Matched position of the object in the scene, or null if not found.

Feature matching enables full-resolution image search: it detects salient features and finds similar images by matching those features. It is robust to differences in resolution, shape, and rotation, but is relatively slower.

Note that feature computation is also time-consuming. Do not compute object features during every match. If the object image doesn't change, compute once and reuse the feature object.

Below is a simple example. It also exists in the built-in Auto.js Pro examples under “Image & Color Processing - Image/Color Search”, and can be run directly.

// Read object (small) image
let hellokitty = $images.read('./hellokitty.jpg');
// Compute object features
let objectFeatures = $images.detectAndComputeFeatures(hellokitty);
// Request screenshot permission
requestScreenCapture();

// Open a HelloKitty page
$app.openUrl('https://baike.baidu.com/item/Hello%20Kitty/984270')

let n = 3;
for (let i = 0; i < n; i++) {
    sleep(3000);
    let capture = captureScreen();

    // To improve performance, you can tweak `scale` when computing scene features (default 0.5):
    // smaller is faster but may cause over-scaling and wrong matches. If matching can't find correct results,
    // try adjusting e.g. { scale: 1 }. You can also set { region: [...] } to compute features only for a region.
    let sceneFeatures = $images.detectAndComputeFeatures(capture);
    // On the last iteration, draw match details to help debugging (adds overhead)
    let drawMatches = (i === n - 1 ? './matches.jpg' : undefined);
    let result = $images.matchFeatures(sceneFeatures, objectFeatures, \{ drawMatches \});
    // Print result and center point. You can click via click(result.centerX, result.centerY)
    console.log(result, result ? result.center : null);

    // Recycle scene feature object
    sceneFeatures.recycle();

    if (drawMatches) {
        // View matches.jpg in current directory (contains detailed match visualization)
        app.viewFile('./matches.jpg');
    }
}
// Recycle object feature object
objectFeatures.recycle();
hellokitty.recycle();

ImageFeatures

[v9.2 Added]

Class that stores feature information. Used only for feature matching.

ImageFeatures.recycle()

Recycles the feature object. You must call this explicitly when you no longer need it; otherwise it may leak memory and crash the app.

ObjectFrame

[v9.2 Added]

Result returned by feature matching, representing a quadrilateral.

ObjectFrame.topLeft

• {Point}

Top-left corner of the quadrilateral.

ObjectFrame.topRight

• {Point}

Top-right corner of the quadrilateral.

ObjectFrame.bottomLeft

• {Point}

Bottom-left corner of the quadrilateral.

ObjectFrame.bottomRight

• {Point}

Bottom-right corner of the quadrilateral.

ObjectFrame.center

• {Point}

Center point of the quadrilateral.

ObjectFrame.centerX

• {number}

X coordinate of the quadrilateral center.

ObjectFrame.centerY

• {number}

Y coordinate of the quadrilateral center.

MatchingResult

[v4.1.0 Added]

MatchingResult.matches

• {Array} Array of match results.

Each element is a Match object:

• point {Point} Match position
• similarity {number} Similarity

Example:

var result = images.matchTemplate(img, template, {
  max: 100,
});
result.matches.forEach((match) => {
  log("point = " + match.point + ", similarity = " + match.similarity);
});

MatchingResult.points

• {Array} Array of match positions.

MatchingResult.first()

• Returns {Match}

First match. Returns null if there are no matches.

MatchingResult.last()

• Returns {Match}

Last match. Returns null if there are no matches.

MatchingResult.leftmost()

• Returns {Match}

Leftmost match in the large image. Returns null if there are no matches.

MatchingResult.topmost()

• Returns {Match}

Topmost match in the large image. Returns null if there are no matches.

MatchingResult.rightmost()

• Returns {Match}

Rightmost match in the large image. Returns null if there are no matches.

MatchingResult.bottommost()

• Returns {Match}

Bottommost match in the large image. Returns null if there are no matches.

MatchingResult.best()

• Returns {Match}

Best (highest similarity) match. Returns null if there are no matches.

MatchingResult.worst()

• Returns {Match}

Worst (lowest similarity) match. Returns null if there are no matches.

MatchingResult.sortBy(cmp)

• cmp {Function} | {string} Compare function, or a string for sort direction. For example, "left" sorts matches left-to-right, "top" sorts top-to-bottom, "left-top" sorts left-to-right then top-to-bottom. Directions include left, top, right, bottom.
• {MatchingResult}

Sort matches and return the sorted result.

var result = images.matchTemplate(img, template, {
  max: 100,
});
log(result.sortBy("top-right"));

Image

Represents an image: a screenshot, a local image, or an image loaded from the network.

Image.getWidth()

Return image width in pixels.

Image.getHeight()

Return image height in pixels.

Image.width

• {number}

Image width (pixels). Same semantics as Image.getWidth().

Image.height

• {number}

Image height (pixels). Same semantics as Image.getHeight().

Image.getPointer()

• Returns {number}

Return the native pointer address of the underlying object (for debugging / low-level investigation only).

Image.getBitmap()

• Returns {Bitmap}

Return the Android Bitmap for this image.

Image.getMat()

• Returns {Mat}

Return the OpenCV Mat for this image.

Image.clone()

• Returns {Image}

Clone the image and return a new Image. The cloned image must be recycled separately.

Image.recycle()

Release resources held by this image. Do not use the image after recycling.

Image.isRecycled()

• Returns {boolean}

Return whether the image has been recycled.

Image.recycled

• {boolean}

Shortcut property for recycled state. Same semantics as Image.isRecycled().

Image.ensureNotRecycled()

Ensure the image has not been recycled. Throws if it has.

Image.saveTo(path)

• path {string} Path

Save the image to path (overwrites if exists).

Image.pixel(x, y)

• x {number} X coordinate
• y {number} Y coordinate

Return the ARGB value of the pixel at (x, y).

The value format is 0xAARRGGBB (a "32-bit integer", though JavaScript does not distinguish integer/float).

The origin is the top-left of the image. The left edge is the Y axis and the top edge is the X axis.

Point

Object returned by findColor / findImage. Represents a point (coordinates).

Point.x

X coordinate.

Point.y

Y coordinate.

ColorMapping

ColorMapping provides a color-search method based on a precomputed mapping. For repeated color searches on the same screenshot, it can be much faster than images.* functions, but requires initialization.

Note: ColorMapping can only be initialized with a screenshot Image object.

Initialization:

// Request screenshot permission
$images.requestScreenCapture();
// Initialize ColorMapping
let ColorMapping = $colors.mapping;
// Create ColorMapping instance
let cm = new ColorMapping();
// Capture
let img = $images.captureScreen();
// Initialize mapping
cm.reset(img);
// Recycle when done
cm.recycle();

If you don't want to recycle manually, use the singleton, which will be automatically recycled when the script ends.

// Request screenshot permission
$images.requestScreenCapture();
// Initialize ColorMapping
let ColorMapping = $colors.mapping;
// Create ColorMapping instance
let cm = ColorMapping.singleton;
// Capture
let img = $images.captureScreen();
// Initialize mapping
cm.reset(img);
// Find color
cm.findColor("#ffffff");

ColorMapping.singleton

• {ColorMapping}

Global singleton instance.

ColorMapping.reset(img)

• img {Image} Screenshot image

Re-initialize the color mapping.

ColorMapping.recycle()

Release resources held by the ColorMapping instance.

ColorMapping.findColor(color[, options])

• color {number} | {string} Color to find
• options {Object} Options:
  • region {Array} Search region (2 or 4 elements). (region[0], region[1]) is top-left; region[2]*region[3] is width/height. If only 2 elements are provided, the region is from (region[0], region[1]) to bottom-right of the image. If omitted, searches the whole image.
  • similarity {number} Similarity threshold in range 0~1 (larger = more similar). 1 means exact color match; 0 means any color matches.
  • threshold {number} Threshold in range 0~255 (smaller = more similar). 0 means exact; 255 means any. Default 4. Convert to float similarity via similarity = (255 - threshold) / 255. Use either similarity or threshold; if both are set, similarity takes precedence.
• Returns {Point}

Find color in the image. Returns a Point if found, otherwise null.

Example: repeated color searching on the same image:

// Request screenshot permission
$images.requestScreenCapture();
// Initialize ColorMapping
let ColorMapping = $colors.mapping;
// Create ColorMapping instance
let cm = new ColorMapping();
// Repeated searching
while (true) {
  // Capture
  let img = $images.captureScreen();
  // Initialize mapping
  cm.reset(img);
  let p1 = cm.findColor("#ffffff");
  if (p1) {
    // ...
    console.log("White point: " + p1);
    continue;
  }
  let p2 = cm.findColor("#000000");
  if (p2) {
    // ...
    console.log("Black point: " + p2);
    continue;
  }
}
// Release ColorMapping
cm.recycle();

Example: search within a region:

// Request screenshot permission
$images.requestScreenCapture();
// Initialize ColorMapping
let ColorMapping = $colors.mapping;
// Create ColorMapping instance
let cm = new ColorMapping();
// Capture
let img = $images.captureScreen();
// Initialize mapping
cm.reset(img);
// Find color in region: top-left (400, 500), size 300*200, threshold 4
let point = cm.findColor("#00ff00", {
  region: [400, 500, 300, 200],
  threshold: 4,
});
if (point) {
  toast("Found: " + point);
} else {
  toast("Not found");
}
// Release ColorMapping
cm.recycle();

ColorMapping.findMultiColors(firstColor, colors, options)

• firstColor {number} | {string} First point color
• colors {Array} Remaining points relative to the first point. Each element is [x, y, color].
• options {Object} Options:
  • region {Array} Search region (same definition as above)
  • threshold {number} Threshold in range 0~255 (smaller = more similar). Default 4. Convert via similarity = (255 - threshold) / 255.
• Returns {Point}

Multi-point color search. Similar to images.findMultiColors, but extremely fast when repeatedly searching within the same image.

Example: repeated multi-point searching:

// Request screenshot permission
$images.requestScreenCapture();
// Initialize ColorMapping
let ColorMapping = $colors.mapping;
// Create ColorMapping instance
let cm = new ColorMapping();
// Capture
let img = $images.captureScreen();
// Initialize mapping
cm.reset(img);
// Multi-point search
let p1 = cm.findMultiColors("#ff00ff", [
  [10, 20, "#ffffff"],
  [30, 40, "#000000"],
]);
let p2 = cm.findMultiColors("#ff00ff", [
  [10, 20, "#ffffff"],
  [30, 40, "#000000"],
]);
log("p1" + p1 + "p2" + p2);
// Release ColorMapping
cm.recycle();

ColorMapping.findAllPointsForColor(color, options)

• color {number} | {string} Color to find
• options {Object} Options:
  • region {Array} Search region (same definition as above)
  • similarity {number} Similarity threshold in range 0~1
  • threshold {number} Threshold in range 0~255. Default 4. Convert via similarity = (255 - threshold) / 255. Use either; if both exist, similarity wins.
• Returns {Array}

Find all points in the image whose color matches color. Returns an array of Point or null if none is found.

Example: find all white points and black points:

// Request screenshot permission
$images.requestScreenCapture();
// Initialize ColorMapping
let ColorMapping = $colors.mapping;
// Create ColorMapping instance
let cm = new ColorMapping();
// Capture
let img = $images.captureScreen();
// Initialize mapping
cm.reset(img);
// Find points
let whitePoints = cm.findAllPointsForColor("#ffffff");
let blackPoints = cm.findAllPointsForColor("#000000");
if (whitePoints != null) {
  log("White points: " + whitePoints.length);
} else {
  log("No white points found");
}
if (blackPoints != null) {
  log("Black points: " + blackPoints.length);
} else {
  log("No black points found");
}
// Release ColorMapping
cm.recycle();