1Media Process Platform (MPP) module directory description: 2 3MPP : Media Process Platform 4MPI : Media Process Interface 5HAL : Hardware Abstract Layer 6OSAL : Operation System Abstract Layer 7 8Rules: 91. header file arrange rule 10a. inc directory in each module folder is for external module usage. 11b. module internal header file should be put along with the implement file. 12c. header file should not contain any relative path or absolute path, all 13 include path should be keep in Makefile. 142. compiling system rule 15a. for cross platform compiling use cmake as the compiling management system. 16b. use cmake out-of-source build, final binary and library will be install to 17 out/ directory. 183. header file include order 19a. MODULE_TAG 20b. system header 21c. osal header 22d. module header 23 24NOTE: 251. Windows support is NOT maintained any more. 262. Mpp support all rockchip chipset now including: 27 RK29XX/RK30XX/RK31XX 28 RK3288/RK3368/RK3399 29 RK3228/RK3229/RK3228H/RK3328 30 RK3566/RK3568 31 RK3588 32 RV1108/RV1107 33 RV1109/RV1126 343. Mpp support all format hardware can support except VC1. 354. you can get demo about mpp applied to linux and android. 36 Liunx : https://github.com/WainDing/mpp_linux_cpp 37 https://github.com/MUZLATAN/ffmpeg_rtsp_mpp 38 Android : https://github.com/c-xh/RKMediaCodecDemo 395. offical github: https://github.com/rockchip-linux/mpp 40 develop github: https://github.com/HermanChen/mpp 41 develop gitee : https://gitee.com/hermanchen82/mpp 42 43More document can be found at http://opensource.rock-chips.com/wiki_Mpp 44 45---- top 46 | 47 |----- build CMake out-of-source build directory 48 | | 49 | |----- cmake cmake script directory 50 | | 51 | |----- android android build directory 52 | | 53 | |----- linux linux build directory 54 | | 55 | |----- vc10-x86_64 visual studio 2010 on x86_64 build directory 56 | | 57 | |----- vc12-x86_64 visual studio 2013 on x86_64 build directory 58 | 59 |----- doc design documents of mpp 60 | 61 |----- inc header file for external usage, including 62 | platform header and mpi header 63 | 64 |----- mpp Media Process Platform : mpi function private 65 | | implement and mpp infrastructure (vpu_api 66 | | private layer) 67 | | 68 | |----- base base components including MppBuffer, MppFrame, 69 | | MppPacket, MppTask, MppMeta, etc. 70 | | 71 | |----- common video codec protocol syntax interface for both 72 | | codec parser and hal 73 | | 74 | |----- codec all video codec parser, convert stream to 75 | | | protocol structure 76 | | | 77 | | |----- inc header files provided by codec module for 78 | | | external usage 79 | | | 80 | | |----- dec 81 | | | | 82 | | | |----- dummy decoder parser work flow sample 83 | | | | 84 | | | |----- h263 85 | | | | 86 | | | |----- h264 87 | | | | 88 | | | |----- h265 89 | | | | 90 | | | |----- m2v mpeg2 parser 91 | | | | 92 | | | |----- mpg4 mpeg4 parser 93 | | | | 94 | | | |----- vp8 95 | | | | 96 | | | |----- vp9 97 | | | | 98 | | | |----- jpeg 99 | | | 100 | | |----- enc 101 | | | 102 | | |----- dummy encoder controllor work flow sample 103 | | | 104 | | |----- h264 105 | | | 106 | | |----- h265 107 | | | 108 | | |----- jpeg 109 | | 110 | |----- hal Hardware Abstract Layer (HAL): modules used in mpi 111 | | | 112 | | |----- inc header files provided by hal for external usage 113 | | | 114 | | |----- iep iep user library 115 | | | 116 | | |----- pp post-processor user library 117 | | | 118 | | |----- rga rga user library 119 | | | 120 | | |----- deinter deinterlace function module including pp/iep/rga 121 | | | 122 | | |----- rkdec rockchip hardware decoder register generation 123 | | | | 124 | | | |----- h264d generate register file from H.264 syntax info 125 | | | | 126 | | | |----- h265d generate register file from H.265 syntax info 127 | | | | 128 | | | |----- vp9d generate register file from vp9 syntax info 129 | | | 130 | | |----- vpu vpu register generation library 131 | | | 132 | | |----- h263d generate register file from H.263 syntax info 133 | | | 134 | | |----- h264d generate register file from H.264 syntax info 135 | | | 136 | | |----- h265d generate register file from H.265 syntax info 137 | | | 138 | | |----- jpegd generate register file from jpeg syntax info 139 | | | 140 | | |----- jpege generate register file from jpeg syntax info 141 | | | 142 | | |----- m2vd generate register file from mpeg2 syntax info 143 | | | 144 | | |----- mpg4d generate register file from mpeg4 syntax info 145 | | | 146 | | |----- vp8d generate register file from vp8 syntax info 147 | | 148 | |----- legacy generate new libvpu to include old vpuapi path 149 | | and new mpp path 150 | | 151 | |----- test mpp internal video protocol unit test and demo 152 | 153 |----- test mpp buffer/packet component unit test and 154 | mpp/mpi/vpu_api demo 155 | 156 |----- out final release binary output directory 157 | | 158 | |----- bin executable binary file output directory 159 | | 160 | |----- inc header file output directory 161 | | 162 | |----- lib library file output directory 163 | 164 |----- osal Operation System Abstract Layer: abstract layer 165 | | for different operation system 166 | | 167 | |----- allocator supported allocator including Android ion and 168 | | Linux drm 169 | | 170 | |----- android google's android 171 | | 172 | |----- inc osal header file for mpp modules 173 | | 174 | |----- linux mainline linux kernel 175 | | 176 | |----- windows microsoft's windows 177 | | 178 | |----- test OASL unit test 179 | 180 |----- tools coding style format tools 181 | 182 |----- utils small util functions 183 184 185Here is the mpp implement overall framework: 186 187 +---------------------------------------+ 188 | | 189 | ffmpeg / OpenMax / gstreamer / libva | 190 | | 191 +---------------------------------------+ 192 193 +-------------------- MPP ----------------------+ 194 | | 195 | +-------------------------+ +--------+ | 196 | | | | | | 197 | | MPI / MPP | | | | 198 | | buffer queue manage | | | | 199 | | | | | | 200 | +-------------------------+ | | | 201 | | | | 202 | +-------------------------+ | | | 203 | | | | | | 204 | | codec | | OSAL | | 205 | | decoder / encoder | | | | 206 | | | | | | 207 | +-------------------------+ | | | 208 | | | | 209 | +-----------+ +-----------+ | | | 210 | | | | | | | | 211 | | parser | | HAL | | | | 212 | | recoder | | reg_gen | | | | 213 | | | | | | | | 214 | +-----------+ +-----------+ +--------| | 215 | | 216 +-------------------- MPP ----------------------+ 217 218 +---------------------------------------+ 219 | | 220 | kernel | 221 | RK vcodec_service / v4l2 | 222 | | 223 +---------------------------------------+ 224 225 226 227Here is the Media Process Interface hierarchical structure 228MpiPacket and MpiFrame is the stream I/O data structure. 229And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and 230Android's ion. 231This part is learned from ffmpeg. 232 233 +-------------------+ 234 | | 235 | MPI | 236 | | 237 +---------+---------+ 238 | 239 | 240 v 241 +---------+---------+ 242 | | 243 +---+ ctx +---+ 244 | | | | 245 | +-------------------+ | 246 | | 247 v v 248 +-------+-------+ +-------+-------+ 249 | | | | 250 | packet | | frame | 251 | | | | 252 +---------------+ +-------+-------+ 253 | | 254 | | 255 | | 256 | +---------------+ | 257 | | | | 258 +---->+ buffer +<----+ 259 | | 260 +---------------+ 261 262 263 264Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer, 265MPP will send the stream to codec layer, codec layer parses the stream header 266and generates a protocol standard output. This output will be send to HAL to 267generate register file set and communicate with hardware. Hardware will complete 268the task and resend information back. MPP notify codec by hardware result, codec 269output decoded frame by display order. 270 271MPI MPP decoder parser HAL 272 273 + + + + + 274 | | | | | 275 | open context | | | | 276 +----------------> | | | | 277 | | | | | 278 | init | | | | 279 +----------------> | | | | 280 | | | | | 281 | | init | | | 282 | +----------------> | | | 283 | | | | | 284 | | | init | | 285 | | +----------------> | | 286 | | | | | 287 | | | | open | 288 | | +-----------------------------------> | 289 | | | | | 290 | decode | | | | 291 +----------------> | | | | 292 | | | | | 293 | | send_stream | | | 294 | +----------------> | | | 295 | | | | | 296 | | | parse_stream | | 297 | | +----------------> | | 298 | | | | | 299 | | | | reg generation | 300 | | +-----------------------------------> | 301 | | | | | 302 | | | | send_regs | 303 | | +-----------------------------------> | 304 | | | | | 305 | | | | wait_regs | 306 | | +-----------------------------------> | 307 | | | | | 308 | | | notify_hw_end | | 309 | | +----------------> | | 310 | | | | | 311 | | get_picture | | | 312 | +----------------> | | | 313 | | | | | 314 | | | get_picture | | 315 | | +----------------> | | 316 | | | | | 317 | flush | | | | 318 +----------------> | | | | 319 | | | | | 320 | | flush | | | 321 | +----------------> | | | 322 | | | | | 323 | | | reset | | 324 | | +----------------> | | 325 | | | | | 326 | close | | | | 327 +----------------> | | | | 328 | | | | | 329 | | close | | | 330 | +----------------> | | | 331 | | | | | 332 | | | close | | 333 | | +----------------> | | 334 | | | | | 335 | | | | close | 336 | | +-----------------------------------> | 337 + + + + + 338 339There are three memory usage modes that decoder can support: 340 341Mode 1: Pure internal mode 342In the mode user will NOT call MPP_DEC_SET_EXT_BUF_GROUP control to decoder. 343Only call MPP_DEC_SET_INFO_CHANGE_READY to let decoder go on. Then decoder will 344use create buffer internally and user need to release each frame they get. 345 346Advantage: 347Easy to use and get a demo quickly 348Disadvantage: 3491. The buffer from decoder may not be return before decoder is close. 350 So memory leak or crash may happen. 3512. The decoder memory usage can not be control. Decoder is on a free-to-run 352 status and consume all memory it can get. 3533. Difficult to implement zero-copy display path. 354 355Mode 2: Half internal mode 356This is the mode current mpi_dec_test code using. User need to create 357MppBufferGroup according to the returned info change MppFrame. 358User can use mpp_buffer_group_limit_config to limit decoder memory usage. 359 360Advantage: 3611. Easy to use 3622. User can release MppBufferGroup after decoder is closed. 363 So memory can stay longer safely. 3643. Can limit the memory usage by mpp_buffer_group_limit_config 365Disadvantage: 3661. The buffer limitation is still not accurate. Memory usage is 100% fixed. 3672. Also difficult to implement zero-copy display path. 368 369Mode 3: Pure external mode 370In this mode use need to create empty MppBufferGroup and import memory from 371external allocator by file handle. 372On Android surfaceflinger will create buffer. Then mediaserver get file handle 373from surfaceflinger and commit to decoder's MppBufferGroup. 374 375Advantage: 3761. Most efficient way for zero-copy display 377Disadvantage: 3781. Difficult to learn and use. 3792. Player work flow may limit this usage. 3803. May need external parser to get the correct buffer size for the external 381 allocator. 382 383The required buffer size caculation: 384hor_stride * ver_stride * 3 / 2 for pixel data 385hor_stride * ver_stride / 2 for extra info 386Total hor_stride * ver_stride * 2 will be enough. 387 388For H.264/H.265 20+ buffers will be enough. 389For other codec 10 buffers will be enough. 390