Mustafa Can Yücel
blog-post-17

Trying to Get OpenCV and CameraX to Play Nice on Android: A Comedy of Errors in a World of Outdated Tutorials

Welcome to the wild world of Android development, where integrating cutting-edge technologies like OpenCV and CameraX can sometimes feel like navigating a maze of outdated tutorials and conflicting dependencies. In this comedic journey, we'll embark on the quest to bring the power of OpenCV's image processing capabilities to Android applications, all while wrestling with the quirks of CameraX and the ever-changing landscape of development tools. Buckle up and prepare for a rollercoaster ride filled with unexpected twists, hilarious mishaps, and ultimately, triumphant solutions.

This post includes on setting up OpenCV on Android, without JNI support (i.e. no C++, only Kotlin).

The versions are:

  • OpenCV: 4.9.0
  • CameraX: 1.3.1
  • Android Studio: 2023.1.1 Patch 2

Assumptions

  • You are using Android Studio as IDE.

Setup

Download the Android SDK

  1. Download the latest version from the official website. It is usually found here.
  2. Extract it into a suitable location. If you are planning to use shared mode where multiple projects just reference to a single SDK on the disk (instead of every project including its own SDK files), remember this location. Actually, remember this location anyway. Note that the SDK is ~200 MB.
  3. The extracted API should have the following directory structure: \OpenCV-android-sdk\sdk.

Adding SDK to Existing Android Studio Project as Module

Compatibility with Gradle Language

The newer Android Studio allows to select a build configuration language from the following:

  • Kotlin DSL (build.gradle.kts)
  • Kotlin DSL (build.gradle.kts) + Gradle Version Catalogs
  • Groovy DSL (build.gradle)

Even though the first one is recommended, OpenCV will not work as a module unless you have selected the third option. The error will be due to the following lines in the build.gradle file:

plugins {
  id 'com.android.application' version '8.2.2' apply false
  id 'org.jetbrains.kotlin.android' version '1.9.22' apply false
}

The Gradle will complain about not being able to find application plugin, and then it will complain about the versions...etc. The easiest solution is creating the project with Groovy DSL for now.

One SDK Per Project

  1. Import the SDK to the existing project as a module using Menu > File > New > New Module > Import... (Note that Import button is at bottom left, separate from the templates).
  2. Select the \OpenCV-android-sdk\sdk as the source directory, and change the module name (e.g. :opencv)

Shared SDK

  1. Add the following line to the settings.gradle file:
def opencvsdk='<path_to_opencv_android_sdk_rootdir>'

include ':opencv'
project(':opencv').projectDir = new File(opencvsdk + '/sdk')
  1. Alternatively, add the following line to the gradle.properties file:
opencvsdk=<path_to_opencv_android_sdk_rootdir>

Adding Dependency to the Module

The easiest way is to open the module settings of your app (F4), then go to Dependencies, select Add Module and select the opencv module.

Alternatively, you can all the following lines to the app/build.gradle file:

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    ...
    implementation project(':opencv')
}

Loading the Library SDK

Add the following line before using any functionality:

System.loadLibrary("opencv_java4")

Handling JavaVersionCompatibility Error

It is very likely that the first build of the application will fail due to the following error:

ERROR : 'compileJava' task (current target is 11) and 'compileKotlin' task (current target is 1.8) jvm target compatibility should be set to the same Java version.

Inconsistent JVM-target compatibility detected for tasks 'compileDebugJavaWithJavac' (1.8) and 'compileDebugKotlin' (17).

To fix this problem, set all the java versions to 17, including the ones in the build.gradle file of the opencv module:

compileOptions {
    sourceCompatibility JavaVersion.VERSION_17
    targetCompatibility JavaVersion.VERSION_17
}

kotlinOptions {
    jvmTarget = '17'
}

Now the OpenCV library SDK should be working; you should be able to build the project with Ctrl+F9

Using CameraX with OpenCV

The camera object (with its UI counterpart) of the OpenCV SDK is really bad; it does not allow any customization and used to have big issues with the device orientation.

What is CameraX?

Android CameraX is a powerful and user-friendly library introduced by Google, designed to simplify the process of integrating camera functionalities into Android applications. It offers a consistent and robust API that abstracts away the complexities of dealing with different device implementations and versions of the Android operating system. CameraX provides developers with a streamlined way to access core camera features, enabling them to focus more on building innovative camera-based experiences rather than dealing with low-level camera APIs.

What is the catch?

The catch is that CameraX uses YUV420-888 format whereas OpenCV deals with BGR8888. Moreover, the chroma planes may be interleaved or ordered. This requires an adapter or a converter between the two libraries.

Converting CameraX Stream to Mat Instances

The ImageAnalysis.Analyzer class allows to intercept the camera stream in the form of an ImageProxy. This is a crucial intermediary for accessing and processing images captured by the device’s camera in real-time, aligned with the lifecycle management of the app. An ImageProxy instance has a property named Image, which if the ImageProxy is a wrapper for an android Image, it will return the Image. It is possible for an ImageProxy to wrap something that isn't an Image. If that's the case then it will return null. The returned image should not be closed by the application. Instead it should be closed by the ImageProxy, which happens, for example, on return from the ImageAnalysis.Analyzer function. Destroying the ImageAnalysis will close the underlying android.media.ImageReader. So an Image obtained with this method will behave as such.

The following extension function can be used for the conversion:

fun Image.yuvToRgba(): Mat {
    val rgbaMat = Mat()

    if (format == ImageFormat.YUV_420_888 && planes.size == 3){
        val chromaPixelStride = planes[1].pixelStride

        if (chromaPixelStride == 2) // chroma channels are interleaved
        {
            assert(planes[0].pixelStride == 1)
            assert(planes[2].pixelStride == 2)
            val yPlane = planes[0].buffer
            val uvPlane1 = planes[1].buffer
            val uvPlane2 = planes[2].buffer

            val yMat = Mat(height, width, CvType.CV_8UC1, yPlane)
            val uvMat1 = Mat(height / 2, width / 2, CvType.CV_8UC2, uvPlane1)
            val uvMat2 = Mat(height / 2, width / 2, CvType.CV_8UC2, uvPlane2)
            val addrDiff = uvMat2.dataAddr() - uvMat1.dataAddr()

            if (addrDiff > 0) {
                assert(addrDiff == 1L)
                Imgproc.cvtColorTwoPlane(yMat,uvMat1, rgbaMat, Imgproc.COLOR_YUV2RGBA_NV12)
            } else {
                assert(addrDiff == -1L)
                Imgproc.cvtColorTwoPlane(yMat, uvMat2, rgbaMat, Imgproc.COLOR_YUV2RGBA_NV21)
            }
        }
        else // chroma channels are not interleaved
        {
            val yuvBytes = ByteArray(width * (height + height / 2))
            val yPlane = planes[0].buffer
            val uPlane = planes[1].buffer
            val vPlane = planes[2].buffer

            yPlane.get(yuvBytes, 0, width * height)

            val chromaRowStride = planes[1].rowStride
            val chromaRowPadding = chromaRowStride - width / 2

            var offset = width * height

            if (chromaRowPadding == 0) {
                // When the row stride of the chroma channels equals their width, we can copy
                // the entire channels in one go
                uPlane.get(yuvBytes, offset, width * height / 4)
                offset += width * height / 4
                vPlane.get(yuvBytes, offset, width * height / 4)
            } else {
                // When not equal, we need to copy the channels row by row
                for (i in 0 until height / 2) {
                    uPlane.get(yuvBytes, offset, width / 2)
                    offset += width / 2
                    if (i < height / 2 - 1) {
                        uPlane.position(uPlane.position() + chromaRowPadding)
                    }
                }
                for (i in 0 until height / 2) {
                    vPlane.get(yuvBytes, offset, width / 2)
                    offset += width / 2
                    if (i < height / 2 - 1) {
                        vPlane.position(vPlane.position() + chromaRowPadding)
                    }
                }
            }

            val yuvMat = Mat(height + height / 2, width, CvType.CV_8UC1)
            yuvMat.put(0,0,yuvBytes)
            Imgproc.cvtColor(yuvMat, rgbaMat, Imgproc.COLOR_YUV2RGBA_I420, 4)
        }
    }
    return rgbaMat
}

Intercepting Frames of CameraX

One of the major advantages of CameraX is being able to bind analyzers to the life cycle of its camera provider. The framework exposes every frame to these analyzers in the form of an ImageProxy. These analyzer classes have to implement the ImageAnalysis.Analyzer interface, which ensures that the class has to contain a function named analyze(image: ImageProxy). Combining this function with the yuvToRgba() extension method defined above, it is possible to process every frame with OpenCV. One important point to note is that since CameraX runs on its own executor, it is not possible to modify the UI elements directly (remember that UI elements can only be accessed from the Main dispatcher). Also, the analyze function is in a separate class, therefore it does not have access to the context of the Activity or Fragment that it is called, making runOnUIThread calls impossible. Since it is a very very bad practice to move the context around, one of the ways to go is defining a listener delegate type that we can give to the analyzer class as a constructor argument. This delegate can be bound to a function on the calling class (Activity or Fragment), and do all the work using the data that is supplied by the analyzer class.

Below is an example analyzer class together with its listener definition that converts the current frame to grayscale:

typealias OpencvListener = (message: String, bitmap: Bitmap) -> Unit

private class LightnessAnalyzer(private val listener: OpencvListener) : ImageAnalysis.Analyzer {
    fun Image.yuvToRgba(): Mat {
        // this method is already explained above
    }

    @OptIn(ExperimentalGetImage::class) override fun analyze(image: ImageProxy) {
        image.image?.let {
            if (it.format == ImageFormat.YUV_420_888 && it.planes.size == 3) {
                val rgbMat = it.yuvToRgba()
                val buf = Mat()
                Imgproc.cvtColor(rgbMat, buf, Imgproc.COLOR_RGBA2GRAY)
                val bmp = Bitmap.createBitmap(buf.cols(), buf.rows(), Bitmap.Config.ARGB_8888)
                val message = "You can pass in additional metadata here"

                Utils.matToBitmap(buf, bmp)

                listener(message, bmp)
            }
        }

        image.close()
    }
}

Note the @OptIn(ExperimentalGetImage::class) attribute; it is required because ImageProxy.image getter is still experimental. This attribute will possibly be obsolete with the future versions.

In the above analyzer class, the listener will be supplied with a message (string) and the grayscale image of the frame as Bitmap every frame.

This listener can be subscribed to within the configuration of the CameraX:

// initialization - see below

// analyze
val opencvAnalyzer = ImageAnalysis.Builder()
      .build()
      .also {
      it.setAnalyzer(cameraExecutor, LightnessAnalyzer { message, bitmap ->
                // Log.d(TAG, message)
                runOnUiThread {
                    // Rotate the bitmap based on the device orientation
                    try {
                        val rotation = when (binding.viewFinder.display.rotation) {
                            Surface.ROTATION_0 -> 90
                            Surface.ROTATION_90 -> 0
                            Surface.ROTATION_180 -> 270
                            Surface.ROTATION_270 -> 180
                            else -> 0
                        }

                        if (rotation == 0) {
                            binding.opencvImage.setImageBitmap(bitmap)
                        } else {
                            val matrix = android.graphics.Matrix()
                            matrix.postRotate(rotation.toFloat())
                            val rotatedBitmap = Bitmap.createBitmap(
                                bitmap,
                                0,
                                0,
                                bitmap.width,
                                bitmap.height,
                                matrix,
                                true
                            )
                            binding.opencvImage.setImageBitmap(rotatedBitmap)
                        }
                    } catch (ex: Exception) {
                        binding.opencvImage.setImageBitmap(bitmap)
                    }
                }
            })
        }

    // back camera as default
    val cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA

    try {
        cameraProvider.unbindAll()

        cameraProvider.bindToLifecycle(
            this, cameraSelector, preview, imageCapture, opencvAnalyzer
        )
    } catch (ex: Exception) {
        Log.e(TAG, "use case binding failed", ex)
    }
}, ContextCompat.getMainExecutor(this))

The above code block gets the grayscale frame bitmap and assigns it to an ImageView element in the UI (opencvImage) through the view binding class (binding) in the UI thread. Note the additional code in the delegate that rotates the image; the reason of this section is that CameraX returns the image 90 degrees rotated CW, or directly as it acquires from the sensor. This means that when the device is in portrait mode, the image has to be rotated, but when the device is in landscape mode, we do not need a transformation. The easiest solution to this problem is getting the orientation of the viewFinder control, which is of type androidx.camera.view.PreviewView. The preview of the CameraX is bound to this control, and it allows us to get the orientation of the device without meddling with the device sensors. One problem with this approach is, during the orientation switching, all the UI elements and non-lifecycle bound components will be destroyed and recreated. However, the camera will continue shooting since it works on a separate executor, resulting in an exception thrown in binding.viewFinder.display.rotation. For this reason, we wrap it in a try-catch block and just push the untouched image within this few transition frames.

The Full CameraX Configuration

Since it is a very flexible and extensible library, CameraX requires slightly more configuration than a camera intent. The following example demonstrates a configuration for the following features:

  • Permissions: It handles the necessary permissions for capturing photos. It also includes capturing audio for video capture, but it is not implemented.
  • Preview: The camera feed is continuously fed to the preview control in the UI, simulating a view-finder.
  • Photo Capture: When the user clicks a button, a photo is captured and saved to the device; it is also added to the media library with metadata.
  • Analyze: The analyzer class creates a grayscale version of the camera feed, and it is displayed on the UI in a smaller ImageView. It uses OpenCV for this.
  • Everything is in an Activity.

For full project see the GitHub repository.

16 July 2023 | Mustafa Can Yücel | OpenCV, CameraX, Android