Contents
- What is a Toast Message?
- Drawbacks of Appium?
- What is OCR?
- What is Tesseract API?
- How to integrate Tesseract API with Appium?
What is a Toast Message
A toast provides simple feedback about an operation in a small popup. It is displayed quickly and disappears after sometime.
Once you set an alarm, a success message appears at the bottom of the screen. In the above diagram the message – This alarm is set for 10 hours and 7 minutes from now. is an example of Toast Message.
Drawbacks of Appium
These toast messages cannot be automated by using Appium. Because, the Android uiautomatorviewer cannot recognize the toast messages.
What is OCR
Optical Character Recognition, or OCR, is a technology that enables you to convert different types of documents, such as scanned paper documents, PDF files or images captured by a digital camera into editable and searchable data.
So the only way to automate toast messages is take a screen shot with the toast message and use any text recognition library (OCR) in order to get the toast message text.
What is Tesseract API
We used Tesseract API for recognizing the text on a image. Tesseract works on Linux, Windows and Mac OS. One drawback with Tesseract is it cannot recognize the handwriting. As this drawback doesn’t hamper our task so we can go a head and use this API.
How to integrate Tesseract API with Appium
- Include the below dependency in your maven project.
|
1 2 3 4 5 |
<dependency> <groupId>org.bytedeco.javacpp-presets</groupId> <artifactId>tesseract</artifactId> <version>3.03-rc1-0.11</version> </dependency> |
2. Take a Screen shot of the page containing the toast message and place it in your hard disk. (Here I created a folder with name toastmessages and saved the image with name toastmessage1.png file.)
we need to validate the text – Added To Cart in the above toast message.
- Create another folder with name tessdata and place the ENG.traineddata file in that folder. This ENG.traineddata(dowload it from https://github.com/tesseract-ocr/tessdata) file contains the various fonts supported by Tesseract API. At present it can recognize upto 64 different fonts of English text.
Place the folder tessdata under class path of the project.
Use the below methods to work with tesseract.
Methods:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
public String getToastMessage() { String filePath=System.getProperty("user.dir")+"\\toastmessages"; File file = new File(filePath); file.mkdir(); sleep(2000l); captureScreenshot(filePath); String str = ""; BytePointer outText; TessBaseAPI api = new TessBaseAPI(); if (api.Init(".", "ENG") != 0) { System.err.println("Could not initialize tesseract."); System.exit(1); } PIX image = pixRead(filePath+"\\toastmessage1.png"); api.SetImage(image); // Get OCR result outText = api.GetUTF8Text(); str = outText.getString(); Assert.assertTrue(!str.isEmpty()); System.out.println("OCR output:\n" + str); // Destroy used object and release memory api.End(); outText.deallocate(); pixDestroy(image); return str; } |
|
1 2 3 4 5 6 7 8 9 10 |
public void captureScreenshot(String path) { File scrFile = ((TakesScreenshot) driver) .getScreenshotAs(OutputType.FILE); try { String filePath=path+"\\toastmessage1.png"; FileUtils.copyFile(scrFile, new File(filePath)); } catch (IOException e) { e.printStackTrace(); } } |
Test
|
1 |
Assert.assertTrue(mePage.getToastMessage().contains("Added To Cart"), "Invalid Toast Message"); |
Output of the getToastMessage() method is
® [.1 Q “II/é‘: _.|| 88% I} 11:56 PM
e Q ‘,5.’ s
Micromax 5005220MHD 127 cm ( 50) Full Q?
HD LED Television With 1 + 2 Year
Exte , 0
“Micromax 50052…” Added To Cart ‘
Rs.
Hi Anish,
Im getting below error when i tried to read text from screenshot using ocr
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000059cd94b2, pid=4344, tid=0x0000000000001b9c
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C [libtesseract-3.dll+0x1994b2]
#
Can i see the project structure, especially when placing tessdata folder?
I got
java.lang.UnsatisfiedLinkError: no jnilept in java.library.path
Caused by: java.lang.UnsatisfiedLinkError: no liblept in java.library.path
when running this code
tesseract.TessBaseAPI api = new tesseract.TessBaseAPI();