1Media Process Platform (MPP) module directory description: 2 3MPP : Media Process Platform 4MPI : Media Process Interface 5HAL : Hardware Abstract Layer 6OSAL : Operation System Abstract Layer 7 8Rules: 91. header file arrange rule 10a. inc directory in each module folder is for external module usage. 11b. module internal header file should be put along with the implement file. 12c. header file should not contain any relative path or absolute path, all 13 include path should be keep in Makefile. 142. compiling system rule 15a. for cross platform compiling use cmake as the compiling management system. 16b. use cmake out-of-source build, final binary and library will be install to 17 out/ directory. 183. header file include order 19a. MODULE_TAG 20b. system header 21c. osal header 22d. module header 23 24NOTE: 251. Windows support is NOT maintained any more. 262. Mpp support all rockchip chipset now including: 27 RK29XX/RK30XX/RK31XX 28 RK3288/RK3368/RK3399 29 RK3228/RK3229/RK3228H/RK3328 30 RK3528/RK3528A 31 RK3562 32 RK3566/RK3568 33 RK3588 34 RV1108/RV1107 35 RV1109/RV1126 363. Mpp support all format hardware can support except VC1. 374. you can get demo about mpp applied to linux and android. 38 Liunx : https://github.com/WainDing/mpp_linux_cpp 39 https://github.com/MUZLATAN 40 https://github.com/nyanmisaka 41 Android : https://github.com/c-xh/RKMediaCodecDemo 425. offical github: https://github.com/rockchip-linux/mpp 43 develop github: https://github.com/HermanChen/mpp 44 develop gitee : https://gitee.com/hermanchen82/mpp 456. Commit message format should base on https://keepachangelog.com/en/1.0.0/ 46 47More document can be found at http://opensource.rock-chips.com/wiki_Mpp 48 49---- top 50 | 51 |----- build CMake out-of-source build directory 52 | | 53 | |----- cmake cmake script directory 54 | | 55 | |----- android android build directory 56 | | 57 | |----- linux linux build directory 58 | | 59 | |----- vc10-x86_64 visual studio 2010 on x86_64 build directory 60 | | 61 | |----- vc12-x86_64 visual studio 2013 on x86_64 build directory 62 | 63 |----- doc design documents of mpp 64 | 65 |----- inc header file for external usage, including 66 | platform header and mpi header 67 | 68 |----- mpp Media Process Platform : mpi function private 69 | | implement and mpp infrastructure (vpu_api 70 | | private layer) 71 | | 72 | |----- base base components including MppBuffer, MppFrame, 73 | | MppPacket, MppTask, MppMeta, etc. 74 | | 75 | |----- common video codec protocol syntax interface for both 76 | | codec parser and hal 77 | | 78 | |----- codec all video codec parser, convert stream to 79 | | | protocol structure 80 | | | 81 | | |----- inc header files provided by codec module for 82 | | | external usage 83 | | | 84 | | |----- dec 85 | | | | 86 | | | |----- dummy decoder parser work flow sample 87 | | | | 88 | | | |----- h263 89 | | | | 90 | | | |----- h264 91 | | | | 92 | | | |----- h265 93 | | | | 94 | | | |----- m2v mpeg2 parser 95 | | | | 96 | | | |----- mpg4 mpeg4 parser 97 | | | | 98 | | | |----- vp8 99 | | | | 100 | | | |----- vp9 101 | | | | 102 | | | |----- jpeg 103 | | | 104 | | |----- enc 105 | | | 106 | | |----- dummy encoder controllor work flow sample 107 | | | 108 | | |----- h264 109 | | | 110 | | |----- h265 111 | | | 112 | | |----- jpeg 113 | | 114 | |----- hal Hardware Abstract Layer (HAL): modules used in mpi 115 | | | 116 | | |----- inc header files provided by hal for external usage 117 | | | 118 | | |----- iep iep user library 119 | | | 120 | | |----- pp post-processor user library 121 | | | 122 | | |----- rga rga user library 123 | | | 124 | | |----- deinter deinterlace function module including pp/iep/rga 125 | | | 126 | | |----- rkdec rockchip hardware decoder register generation 127 | | | | 128 | | | |----- h264d generate register file from H.264 syntax info 129 | | | | 130 | | | |----- h265d generate register file from H.265 syntax info 131 | | | | 132 | | | |----- vp9d generate register file from vp9 syntax info 133 | | | 134 | | |----- vpu vpu register generation library 135 | | | 136 | | |----- h263d generate register file from H.263 syntax info 137 | | | 138 | | |----- h264d generate register file from H.264 syntax info 139 | | | 140 | | |----- h265d generate register file from H.265 syntax info 141 | | | 142 | | |----- jpegd generate register file from jpeg syntax info 143 | | | 144 | | |----- jpege generate register file from jpeg syntax info 145 | | | 146 | | |----- m2vd generate register file from mpeg2 syntax info 147 | | | 148 | | |----- mpg4d generate register file from mpeg4 syntax info 149 | | | 150 | | |----- vp8d generate register file from vp8 syntax info 151 | | 152 | |----- legacy generate new libvpu to include old vpuapi path 153 | | and new mpp path 154 | | 155 | |----- test mpp internal video protocol unit test and demo 156 | 157 |----- test mpp buffer/packet component unit test and 158 | mpp/mpi/vpu_api demo 159 | 160 |----- out final release binary output directory 161 | | 162 | |----- bin executable binary file output directory 163 | | 164 | |----- inc header file output directory 165 | | 166 | |----- lib library file output directory 167 | 168 |----- osal Operation System Abstract Layer: abstract layer 169 | | for different operation system 170 | | 171 | |----- allocator supported allocator including Android ion and 172 | | Linux drm 173 | | 174 | |----- android google's android 175 | | 176 | |----- inc osal header file for mpp modules 177 | | 178 | |----- linux mainline linux kernel 179 | | 180 | |----- windows microsoft's windows 181 | | 182 | |----- test OASL unit test 183 | 184 |----- tools coding style format tools 185 | 186 |----- utils small util functions 187 188 189Here is the mpp implement overall framework: 190 191 +---------------------------------------+ 192 | | 193 | OpenMax / gstreamer / libva | 194 | | 195 +---------------------------------------+ 196 197 +-------------------- MPP ----------------------+ 198 | | 199 | +-------------------------+ +--------+ | 200 | | | | | | 201 | | MPI / MPP | | | | 202 | | buffer queue manage | | | | 203 | | | | | | 204 | +-------------------------+ | | | 205 | | | | 206 | +-------------------------+ | | | 207 | | | | | | 208 | | codec | | OSAL | | 209 | | decoder / encoder | | | | 210 | | | | | | 211 | +-------------------------+ | | | 212 | | | | 213 | +-----------+ +-----------+ | | | 214 | | | | | | | | 215 | | parser | | HAL | | | | 216 | | recoder | | reg_gen | | | | 217 | | | | | | | | 218 | +-----------+ +-----------+ +--------| | 219 | | 220 +-------------------- MPP ----------------------+ 221 222 +---------------------------------------+ 223 | | 224 | kernel | 225 | RK vcodec_service / v4l2 | 226 | | 227 +---------------------------------------+ 228 229 230 231Here is the Media Process Interface hierarchical structure 232MpiPacket and MpiFrame is the stream I/O data structure. 233And MpiBuffer encapsulates different buffer implement like Linux's dma-buf and 234Android's ion. 235This part is learned from ff. 236 237 +-------------------+ 238 | | 239 | MPI | 240 | | 241 +---------+---------+ 242 | 243 | 244 v 245 +---------+---------+ 246 | | 247 +---+ ctx +---+ 248 | | | | 249 | +-------------------+ | 250 | | 251 v v 252 +-------+-------+ +-------+-------+ 253 | | | | 254 | packet | | frame | 255 | | | | 256 +---------------+ +-------+-------+ 257 | | 258 | | 259 | | 260 | +---------------+ | 261 | | | | 262 +---->+ buffer +<----+ 263 | | 264 +---------------+ 265 266 267 268Take H.264 deocder for example. Video stream will first queued by MPI/MPP layer, 269MPP will send the stream to codec layer, codec layer parses the stream header 270and generates a protocol standard output. This output will be send to HAL to 271generate register file set and communicate with hardware. Hardware will complete 272the task and resend information back. MPP notify codec by hardware result, codec 273output decoded frame by display order. 274 275MPI MPP decoder parser HAL 276 277 + + + + + 278 | | | | | 279 | open context | | | | 280 +----------------> | | | | 281 | | | | | 282 | init | | | | 283 +----------------> | | | | 284 | | | | | 285 | | init | | | 286 | +----------------> | | | 287 | | | | | 288 | | | init | | 289 | | +----------------> | | 290 | | | | | 291 | | | | open | 292 | | +-----------------------------------> | 293 | | | | | 294 | decode | | | | 295 +----------------> | | | | 296 | | | | | 297 | | send_stream | | | 298 | +----------------> | | | 299 | | | | | 300 | | | parse_stream | | 301 | | +----------------> | | 302 | | | | | 303 | | | | reg generation | 304 | | +-----------------------------------> | 305 | | | | | 306 | | | | send_regs | 307 | | +-----------------------------------> | 308 | | | | | 309 | | | | wait_regs | 310 | | +-----------------------------------> | 311 | | | | | 312 | | | notify_hw_end | | 313 | | +----------------> | | 314 | | | | | 315 | | get_picture | | | 316 | +----------------> | | | 317 | | | | | 318 | | | get_picture | | 319 | | +----------------> | | 320 | | | | | 321 | flush | | | | 322 +----------------> | | | | 323 | | | | | 324 | | flush | | | 325 | +----------------> | | | 326 | | | | | 327 | | | reset | | 328 | | +----------------> | | 329 | | | | | 330 | close | | | | 331 +----------------> | | | | 332 | | | | | 333 | | close | | | 334 | +----------------> | | | 335 | | | | | 336 | | | close | | 337 | | +----------------> | | 338 | | | | | 339 | | | | close | 340 | | +-----------------------------------> | 341 + + + + + 342 343There are three memory usage modes that decoder can support: 344 345Mode 1: Pure internal mode 346In the mode user will NOT call MPP_DEC_SET_EXT_BUF_GROUP control to decoder. 347Only call MPP_DEC_SET_INFO_CHANGE_READY to let decoder go on. Then decoder will 348use create buffer internally and user need to release each frame they get. 349 350Advantage: 351Easy to use and get a demo quickly 352Disadvantage: 3531. The buffer from decoder may not be return before decoder is close. 354 So memory leak or crash may happen. 3552. The decoder memory usage can not be control. Decoder is on a free-to-run 356 status and consume all memory it can get. 3573. Difficult to implement zero-copy display path. 358 359Mode 2: Half internal mode 360This is the mode current mpi_dec_test code using. User need to create 361MppBufferGroup according to the returned info change MppFrame. 362User can use mpp_buffer_group_limit_config to limit decoder memory usage. 363 364Advantage: 3651. Easy to use 3662. User can release MppBufferGroup after decoder is closed. 367 So memory can stay longer safely. 3683. Can limit the memory usage by mpp_buffer_group_limit_config 369Disadvantage: 3701. The buffer limitation is still not accurate. Memory usage is 100% fixed. 3712. Also difficult to implement zero-copy display path. 372 373Mode 3: Pure external mode 374In this mode use need to create empty MppBufferGroup and import memory from 375external allocator by file handle. 376On Android surfaceflinger will create buffer. Then mediaserver get file handle 377from surfaceflinger and commit to decoder's MppBufferGroup. 378 379Advantage: 3801. Most efficient way for zero-copy display 381Disadvantage: 3821. Difficult to learn and use. 3832. Player work flow may limit this usage. 3843. May need external parser to get the correct buffer size for the external 385 allocator. 386 387The required buffer size caculation: 388hor_stride * ver_stride * 3 / 2 for pixel data 389hor_stride * ver_stride / 2 for extra info 390Total hor_stride * ver_stride * 2 will be enough. 391 392For H.264/H.265 20+ buffers will be enough. 393For other codec 10 buffers will be enough. 394