Sunday 17 September 2017

HI3518 Camera Module - Part 3 - Capturing Images with an HI3518/SC1035 Camera Module

Introduction

This is part 3 of a series of blog posts about a cheap HI3518 based camera module. In part 1, I showed how to build a custom image for the module. In part 2, I fixed a bug that was preventing the Ethernet from working. This part covers capturing an image.

Once I had built a custom image and got the Ethernet working I was ready to compile the HI3518 sample programs that are available in the SDK. To make this easier I set up an NFS share on my development computer and had the camera module automatically mount the share on boot up. This removed the need to include the programs in the image and so removed the need to build a new image each time I wanted to test a change to the programs, or the need to be constantly copying files around.

I was under the impression that the camera module I had contained an AR0130 image sensor. This is what the listing said, but it was also what the camera module reported in its boot logs. This however turned out not to be the case. I tried compiling the sample programs to target the AR0130 but could not get them to work. After a lot of playing around wondering what I was doing wrong I realised that the camera module I have actually contains an SC1035, which has a similar specification to the AR0130 but is not a drop in replacement. And, unfortunately the SDK did not include code for the SC1035.

The image below shows how you can distinguish between the two image sensors by looking at the layout of the contacts.



Comparison of the layout of the contacts of the AR0130 (left) and SC1035 (right)

Reverse Engineering

The following is a relatively lengthy description of the steps I took to figure out the minimum settings needed to capture an image with the SC1035 image sensor. If you want, you can skip this section and go straight to "Testing the Settings".

To be able to capture an image there is a minimum amount of configuration that needs to be known. To keep the amount of configuration that I needed to figure out to a minimum I started by attempting to get the vi_bayerdump sample program provided in the SDK working. This program bypasses the HI3518 ISP (Image Signal Processor) and provides the raw sensor data (in a bayer mosaic format). This removed the need to get all the ISP parts working. This meant that the following information needed to be figured out:
  • The register settings needed to initialise the image sensor
  • The correct values for the structures used to describe the device in the HI3518 API
  • The clock frequency needed by the image sensor
I tried searching the internet but initially didn’t find any useful information. The following are some of the steps I used to determine the correct settings for the module.

All of the code I ended up writing for the steps covered in this blog post is available in the following GitHub repository:

https://github.com/mark4h/HI3518-SC1035

Modifying the I2C driver to log read and write calls

In an attempt to figure out what register settings were being sent to the image sensor over I2C, I modified the hi_i2c driver, availble in the SDK, to log the read and write calls. I modified the I2C_Ioctl function in hii2c.c to add kernel print statements (printk) and then compiled the module and replaced the original version in the preinstalled image on the module. On booting into the original image I was then able to see all the I2C read and write operation being performed. This allowed me to identify which address the SC1035 was on (0x60), the register address length (2 bytes) and the data length (1 byte). It also gave me the registers and values that the module wrote to, however the ISP appear to be bombarding the sensor with a lot of writes so it was very hard to determine what the minimum commands were.

Reverse engineer the Sofia program

The original preinstalled image runs a program called Sofia which is the main program responsible for capturing images and providing a web frontend.

I extracted the Sofia program from the preinstalled image and ran it through the Snowman decompiler (Note: you will need to turn off auto decompilation because decompiling the whole Snowman application is very intensive).

The first thing I noticed was a hint to which sensors this version of Sofia had support for, by the names of 4 objects:
  • gAr0130SensorExpFuncs
  • gH81SensorExpFuncs
  • gOv9750SensorExpFuncs
  • gSc1035SensorExpFuncs
I then tried searching the instructions for the register addresses that I knew the Sofia program was writing to based on the logs from the modified hi_i2c driver. I identified a function that started at the address 0x4c372 and ended at 0x4c3c0c.

Decompiling this gave me what looked like the sensor_init function for the SC1035 sensor.

I wrote a Python script to tidy up the c code generated by Snowman and make it call sensor_write_register with the register address and data value, which is the format used for other sensor initialisation routines in the SDK samples.

Device and Channel settings

The final piece of configuration needed was how to define the VI_DEV_ATTR_S and VI_CHN_ATTR_S structures need by the HI3518 API.

These structures are passed to the HI_MPI_VI_SetDevAttr and HI_MPI_VI_SetChnAttr functions. While searching for these function in the API header files I noticed that equivalent get functions existed:

HI_S32 HI_MPI_VI_SetDevAttr(VI_DEV ViDev, const VI_DEV_ATTR_S *pstDevAttr);
HI_S32 HI_MPI_VI_GetDevAttr(VI_DEV ViDev, VI_DEV_ATTR_S *pstDevAttr);
HI_S32 HI_MPI_VI_EnableDev(VI_DEV ViDev);
HI_S32 HI_MPI_VI_DisableDev(VI_DEV ViDev);
HI_S32 HI_MPI_VI_SetChnAttr(VI_CHN ViChn, const VI_CHN_ATTR_S *pstAttr);
HI_S32 HI_MPI_VI_GetChnAttr(VI_CHN ViChn, VI_CHN_ATTR_S *pstAttr);

So I wrote a basic program to call HI_MPI_VI_GetDevAttr and HI_MPI_VI_GetChnAttr and print out the structures. I compiled this program, added it to the original preinstalled image, burnt the modified image back onto the camera module and booted it. This gave me the final piece of the puzzle.

Datasheets

After getting a working version of the vi_bayerdump for the SC1035, I finally managed to track down some Chinese versions of the datasheets. A datasheet called “SC1035 Design and Applications” includes a basic configuration for the registers. The suggested register values closely matched those I worked out through reverse engineering the Sofia program. However, when I tried using the register settings suggested in the datasheet the program failed to capture an image. When I investigated the difference between the settings I found through reverse engineering and those suggested in the datasheet I found that I need to include setting the register 0x3223 to 0x22.

Camera Clock Signal

One other piece of configuration needed is the clock frequency used to drive the image sensor. The SC1035 can be driven at a range of frequencies, but the frequency chosen affects how some of the image sensors registries need to be set. The settings found above were based on a 27 MHz clock. See the section "Setting the Image Sensors Clock" below for how to manually change the clock when testing the settings.

Testing the Settings

Compiling the Program

To test the settings I modified a version of the vi_bayerdump sample program available in the HI3518 SDK. This can be found in the GitHub repository I've put up.

To build the program you will need to checkout the repository and then run make. You will need to set two variables before running make: CROSS and REL_DIR. REL_DIR should point to the directory containing the lib/ and include/ directories from the SDK, which if you have successfully build the image in blog post 2, can be found in the output directory.

An example of the command that needs running is:

CROSS=/home/mark4h/projects/hi3518/hi35xx-buildroot/output/host/usr/bin/arm-buildroot-linux-uclibcgnueabi- REL_DIR=/home/mark4h/projects/hi3518/hi35xx-buildroot/output/build/himpp-hi3518v100-1.0.A.0 make

To test the program I booted into the image created in blog post 2 of this series.

Setting the Image Sensor's Clock

Before I could run the program I needed to make sure that the clock being used to drive the image sensor was set to the correct frequency. On boot up the setting of the clock is performed in the /etc/init.d/S25himpp script.

In the script /etc/init.d/S25himpp the sensor type is read from the arguments passed to the kernel by uboot. However, this was not correctly set up for the SC1035 image sensor and takes a bit of effort to get working so I instead manually set the clock for this test.

In the S25himpp script, the default sensor is ov9712 which uses clock of 24 MHz

Ideally you would update the script to have an option for SC1035, which would then set the sensor clock to 27 MHz as needed.

However, for this test we can instead run the following command manually to set the clock to 27 MHz:

devmem  0x20030030 32 0x5

Setting up NFS

Assuming you have set up an NFS share on your development computer you can then mount the share on the camera module.

To mount an NFS drive first make sure you have the correct network settings.
e.g.:

vi /etc/network/interfaces

address 192.168.2.2 
network 192.168.2.0
netmask 255.255.255.0
broadcast 192.168.2.255
gateway 192.168.2.1

Restart the network if needed:

/etc/init.d/S40network restart

Then mount the NFS.
e.g:

mkdir /nfs

mount -o rw,nolock 192.168.2.1:/srv/nfs /nfs

Running the example

Once you have compiled the example program, set the image sensors clock to the correct frequency and mounted an NFS, you should be ready to run the program:
e.g.:

cd /nfs/vi_bayerdump_SC1035

./vi_bayerdump_SC1035 12 1

This should have created an image named vi_chn_0_1280_720_1_12bits.raw in the same directory as the executable. This file contains the raw data off of the image sensor. To be able to view the image you need to perform demosaicing.

Raw Data Demosaicing

To view the image, I wrote a basic webpage using Javascript to perform bilinear demosaicing.

This can be found in the repository for this blog post.

Appendix - SC1135

After finishing the steps described in this blog post I came across code in the hi35xx-buildroot repository for an HI3518v200/SC1135 combination. The SC1135 appears to have a lot of commonality with the SC1035.

The following link states that “According to the datasheet of SC1135, the SC1135 is the upgraded version of its predecessor - SC1035 ...”

However, I do not believe that I can make use of this code directly since it if for the v200 version of the HI3518 and my module is based on the v100.

11 comments:

  1. Cool, Mark!! Good work!! Keep it up! This was exactly what I was looking for!! To get something like still capture, I tried also looking at Sofia, SDK examples/doing cross-GCC for own App and so on, but didn't manage to make that working. Looking forward to next holidays, I'll give it a try again!! (have an AR0130 and IMX225) Regards, Lothar

    ReplyDelete
  2. Hi Mark! Thanks a lot for all the information you collected (and your python program to clean the decompiled code).

    Today I managed to obtain the first RAW image from my IMX225 board! Hooray!
    I'm using the Hi3518 SDK V1.0.A.0, which includes code for initializing the IMX225. Thus I modified you bayerdump code to use the init code from the SDK - which, however did never work. I always got a HI_ERR_VI_BUF_EMPTY in the GetFrame().

    I compared the initialization code of the SDK with the sensor datasheet and found a few stange settings - however, I overlooked one wrong setting in the SDK code.

    In the end I followed your steps, fired up Snowman (never used a decompiler before :-) ) and after quite some time found the register setting function calls. (there are 2 code blocks in my Sofia version, seems like one is for the HD720p, one for the QuadVGA mode).

    The error I overlooked in the SDK init code was (of course!) in the very last register setting... Using the value from the decompiled Sofia code for that register did the trick!

    Note that I did not yet use the full init code decompiled from Sofia because (a) the one I obtained by fixing the SDK code using the datasheet tables seems to work and (b) the Sofia code uses quite a few settings that contradict the data sheet.
    (I did not yet demosaic my RAW image, but it looks reasonable when opening with GIMP as file type "raw data", file ending ".data")

    I'm also still using the original firmware (also orig. kernel) with only minor modifications to automatically start telnet, mount an NFS share, deactivate the watchdog, etc...

    If anyone is interested... here's the init code I ended up with:
    https://pastebin.com/q8nMnhpq
    Feel free to use, improve, archive...

    Next (when I find the time) I'll try to get long-exposure and high gain settings running (full manual control) and then mount that super-cheap thing to my telescope...
    The datasheet indicates that the imx225 can also do 2x2 binning, but that's too far for now.

    ReplyDelete
  3. short update:
    Meanwhile I can grab RAW data from the imx225 sensor for the complete range of exposure times as well as gains (analog and digital gain). Control of high/low conversion gain is also functional.
    What I did not yet look into is:
    - 2x2 binning
    - crop in sensor (for higher frame rates or square 960x960 frames)
    Currently I send the sensor data via UDP over the 100mbit eth interface to a laptop. However, with the 1280x720 sensor resolution and 16bit/pixel the frame rate is limited to approx 5 FPS (one 1280x720px16bpp frame=1843200Bytes).
    I once tried to copy the most significant 8bit/px out of the 16 bit/px, but the bitshift+copy operation slowed down the CPU so much that I didn't achieve higher rates in the end. (can be programmed more efficient than I did for sure).

    ReplyDelete
    Replies
    1. do you have github?) It's very interesting. Please, if you not harder write tutorial how you make this, or upload you're uImage.

      Delete
    2. Dear starwatcher,
      I also has hi3518+imx225 and can't obtain any pictures.
      I follow this methods and something went wrong.
      Could you, please, reply to this email hi3518@gmx.com to give me files necessary for imx225 sensor. Some of pastebin information not clear (comments after//)
      Thanks in advance!

      Delete
    3. This ZIP here may help (contains mpp2 folder with examples)
      http://s000.tinyupload.com/index.php?file_id=39869905781089335895
      Open the ZIP with "imx225_mpp2_package" (without quotes).

      Delete
    4. Hi,
      30s exposure done with this board (hi3518+imx225 ).
      It's hard to have a nice pic because the sensor saturates to much. I change the original SPI by an other more bigger.
      I compile all SDK except the Uboot because with my new Uboot, the network won't start.
      starwatcher,I try your samples and it's a good start but with 'raw_stream_tx' the image become more and more lighter so i wrote a similar software.
      I'll post all my work on my website soon (sources (C and Delphi),SPI Dump etc..)

      Delete
  4. This is really good. I am going attempt ot today. I often wonder why is it so hard to get a camera to modify the system in a way that i can setup my own aws feed and ptz control, so i can use it to make my own applications. Using pi is so un reliable and these commercial cameras are so hard to hack.

    ReplyDelete
  5. Hello Mark, you did an awesome work, congratulations.
    I´m working on porting my ANPR algorithm from x86 into a Hi3519 architecture.
    I wanted to send you a private email, but I couldn´t find your address, if you want to answer me, my email is a#n#d#r#e#j#1#0#0# at gmail (remove even chars). Take a look to my website www.visart.com.ar
    Cheers.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. Hello, I found your blog very interesting and similar to my interests. I am wondering how difficult it would be to build a firmware for a FH8812? I have never built a firmware before, and am excited about the possibilities it will open up. Thanks for your blog!

    ReplyDelete