Skip to main content
Docs
Go to the Studio
Roberty Studio
RPAModules

Image Recognition

Image Recognition

The Image Recognition module is responsible for recognizing images on the device screen and performing given actions based on the image recognized. Its methods can be accessed as in the example below:

js
await ImageRecognition.waitFor["v1_0_0"]({
  imagePath: `C:/Users/Roberty/Downloads/teste.png`,
})

Methods​

advancedOcr​

Method responsible for performing advanced optical character recognition on a file, with support for extracting words, lines, tables, forms and structured queries.

js
const result = await ImageRecognition.advancedOcr["v1_0_0"]({
  filePath: 'C:/Users/Roberty/Documents/documento.pdf',
  featureType: ['WORD', 'LINE', 'TABLE'],
})

Required parameters​

  • filePath: String - path of the image or PDF file that will be processed.
  • featureType: String[] - list of content types to be extracted. Values: WORD | LINE | TABLE | FORM | QUERY.

Optional parameters​

  • timeout: Number - maximum time in seconds to wait for the processing. Default value: 120.
  • pageRanges: String - range of pages to process (e.g. 1-3, 2). Applicable only to PDFs.
  • queries: String[] - list of questions for the targeted extraction of information from the document (it requires QUERY in featureType).
  • actionId: String - identifier of the action for execution control.

Return​

The constant or variable created, such as the result of the example shown earlier, receives an object with the following properties:

  • result.words: Object[] - list of words recognized with their positions.
  • result.lines: Object[] - list of lines recognized with their positions.
  • result.queries: Object[] - answers of the queries provided in the queries parameter.
  • result.forms: Object[] - key-value pairs extracted from forms.
  • result.tables: Object[] - tables extracted with their rows and columns.
  • result.content: String - the whole textual content extracted from the document.

captchaNormal​

Action responsible for solving a normal captcha. A normal captcha is an image containing distorted but human-readable text.

js
const captchaNormal = await Captcha.normal["v1_0_0"]({
  img: ''
})

Required parameters​

  • img: String - expects the value of the image, which must be provided in base 64.

Return​

The constant or variable created, such as the captchaNormal of the example, receives the string with the solved captcha text directly.

captchaText​

Action responsible for solving a text captcha. A Text Captcha is a type of captcha represented as text and containing no images. Usually, you need to answer a question to pass the verification.

js
const captchaText = await Captcha.text["v1_0_0"]({
  textCaptcha: ''
})

Required parameters​

  • textCaptcha: String - expects the text of the captcha that needs to be answered.

Return​

The constant or variable created, such as the captchaText of the example, receives the string with the captcha answer directly.

disappear​

Method responsible for waiting for images to disappear from the screen. The method does not finish while all the images have not disappeared. The check is repeated every interval seconds; if none of the images is on the screen, the method finishes immediately.

js
const disappearedImage = await ImageRecognition.disappear["v1_0_0"]({
  imagePath: [`C:/Users/Roberty/Downloads/teste.png`, `C:/Users/Roberty/Downloads/teste2.png`],
})

Required parameters​

  • imagePath: String[] - expects an array of strings with the paths of the images that will be waited for to disappear from the screen.

Optional parameters​

  • options: Object - expects an object with the method options. The options are:
    • region: Object - expects an object with the coordinates of the screen region where the image will be waited for. The coordinates are:
      • x: Number - x coordinate of the region.
      • y: Number - y coordinate of the region.
      • width: Number - width of the region.
      • height: Number - height of the region.
    • threshold: Number - expects a number from 0 to 1 representing the similarity between the image saved and the image being waited for on the screen.
  • interval: Number - time in seconds between each check. Default value: 5.
  • attempts: Number - maximum number of checks in which the images are still on the screen. When this number is reached, the execution is interrupted with an error. If it is not provided, the method waits indefinitely.

Return​

The constant or variable created, such as the disappearedImage of the example shown earlier, receives the value true when the images disappear. If the number of attempts is reached before that, the execution is interrupted with an error.

exists​

Method responsible for checking whether an image exists on the screen or not.

js
const existImage = await ImageRecognition.exists["v1_0_0"]({
  imagePath: `C:/Users/Roberty/Downloads/teste.png`
})

Required parameters​

  • imagePath: String - expects a string with the path of the image that will be checked.

Optional parameters​

  • options: Object - expects an object with the method options. The options are:
    • region: Object - expects an object with the coordinates of the screen region where the image will be checked. The coordinates are:
      • x: Number - x coordinate of the region.
      • y: Number - y coordinate of the region.
      • width: Number - width of the region.
      • height: Number - height of the region.
    • threshold: Number - expects a number from 0 to 1 representing the similarity between the image saved and the image being checked on the screen.

Return​

The constant or variable created, such as the existImage of the example shown earlier, receives a value of the boolean type. If the image exists, the value returned is true; otherwise, the value returned is false.

find​

Method responsible for finding the position of an image on the screen.

js
const foundImage = await ImageRecognition.find["v1_0_0"]({
  imagePath: `C:/Users/Roberty/Downloads/teste.png`
})

Required parameters​

  • imagePath: String - expects a string with the path of the image that will be found.

Optional parameters​

  • options: Object - expects an object with the method options. The options are:
    • region: Object - expects an object with the coordinates of the screen region where the image will be found. The coordinates are:
      • x: Number - x coordinate of the region.
      • y: Number - y coordinate of the region.
      • width: Number - width of the region.
      • height: Number - height of the region.
    • threshold: Number - expects a number from 0 to 1 representing the similarity between the image saved and the image being found on the screen.
    • timeout: Number - time in seconds the method will wait to find the image. Default value: 30.

Return​

The constant or variable created, such as the foundImage of the example shown earlier, receives a value of the Object type. The object returned has the following properties:

  • foundImage.x: Number - x coordinate of the position of the image found.
  • foundImage.y: Number - y coordinate of the position of the image found.
  • foundImage.w: Number - width of the image found.
  • foundImage.h: Number - height of the image found.
  • foundImage.exists: Boolean - indicates whether the image was found or not.
  • foundImage.center: Object - coordinates of the center of the image found:
    • x: Number - x coordinate of the center.
    • y: Number - y coordinate of the center.

findAndClick​

Method responsible for clicking on an image found on the screen.

js
await ImageRecognition.findAndClick["v1_0_0"]({
  imagePath: `C:/Users/Roberty/Downloads/teste.png`
})

Required parameters​

  • imagePath: String - expects a string with the path of the image that will be clicked.

Optional parameters​

  • button: String - expects a string with the button that will be clicked. Values: left | right. Default value: left.
  • times: Number - number of clicks. Default value: 1.
  • options: Object - expects an object with the method options. The options are:
    • region: Object - coordinates of the screen region where the image will be found:
      • x: Number - x coordinate of the region.
      • y: Number - y coordinate of the region.
      • width: Number - width of the region.
      • height: Number - height of the region.
    • threshold: Number - expects a number from 0 to 1 representing the similarity between the image saved and the image being found on the screen.
    • timeout: Number - time in seconds the method will wait to find the image. Default value: 30.

Return​

This method returns no values.

findAndMoveMouse​

Method responsible for moving the mouse to an image found on the screen.

js
await ImageRecognition.findAndMoveMouse["v1_0_0"]({
  imagePath: `C:/Users/Roberty/Downloads/teste.png`
})

Required parameters​

  • imagePath: String - expects a string with the path of the image that will be found on the screen and will receive the mouse action.

Optional parameters​

  • options: Object - expects an object with the method options. The options are:
    • region: Object - coordinates of the screen region where the image will be found:
      • x: Number - x coordinate of the region.
      • y: Number - y coordinate of the region.
      • width: Number - width of the region.
      • height: Number - height of the region.
    • threshold: Number - expects a number from 0 to 1 representing the similarity between the image saved and the image being found on the screen.
    • timeout: Number - time in seconds the method will wait to find the image. Default value: 30.

Return​

This method returns no values.

ocr​

Method responsible for performing the recognition of the characters present in a given image.

js
const ocrImage = await ImageRecognition.ocr["v1_0_0"]({
  image: `C:/Users/Roberty/Downloads/teste.png`,
  lang: `pt`
})

Required parameters​

  • image: String - expects a string with the path of the image that will be recognized.

Optional parameters​

  • lang: String - expects a string with the language that will be used to recognize the characters. Default value: pt. The language options are:
    • pt: Portuguese
    • en: English

Return​

The constant or variable created, such as the ocrImage of the example shown earlier, receives a value of the String type. This string represents the characters recognized in the image.

waitFor​

Method responsible for waiting for an image on the main screen of the device. If the image is not found within the time stipulated (timeout), the execution of the code is interrupted with an error.

js
const awaitedImage = await ImageRecognition.waitFor["v1_0_0"]({
  imagePath: `C:/Users/Roberty/Downloads/teste.png`
})

Required parameters​

  • imagePath: String - expects a string with the path of the image that will be waited for on the screen.

Optional parameters​

  • options: Object - expects an object with the method options. The options are:
    • timeout: Number - waiting time in seconds for the image to be found. Default value: 30.
    • region: Object - coordinates of the screen region where the image will be waited for:
      • x: Number - x coordinate of the region.
      • y: Number - y coordinate of the region.
      • width: Number - width of the region.
      • height: Number - height of the region.
    • threshold: Number - expects a number from 0 to 1 representing the similarity of the image that will be waited for.

Return​

This method has no return.