目前android上，录相大多是mp4的视频，这在一般情况下，已经够用了。但是在一些特定的场景，比如远程临控录相或者行车记录仪上，用mp4录相，就不太理想了。为什么呢？因为远程录相，或者行车记录仪上都有一个共同的问题，那就是录相有可能中断。比如突然撞车了，或者是远程监控断电了，如果这时录的是Mp4的视频，那么就会导致，因为没有来得及写mp4的文件头信息，从而打不开视频。所以在远程监控录相和行车记录仪上，录相的格式，最好使用mpeg2ts流。
现在android无论是8.0还是9.0、10.0上，都支持录制mpeg2ts流视频，但是却不支持用MediaMuxer的writeSampleData去打包mpeg2ts。这就导致了一个问题，比如有的app上，想将一个mp4的视频，转成mpeg2ts流的视频，就无法在java端完成。且现在android8.1(9.0、10.0上没有试)，录下来的mpeg2ts流，经常会丢帧，最后几帧录不下来。这个Mpeg2Ts功能显得很鸡肋。下面，我们就来讨论一下，怎么去解决这些问题。
先说一下mpeg2ts录相丢帧的问题。mpeg2ts录相的framework层cpp文件是frameworks\av\media\libstagefrightMPEG2TSWriter.cpp这一个。写数据的函数是：MPEG2TSWriter::onMessageReceived(const sp<AMessage> &msg) 这个函数：
void MPEG2TSWriter::onMessageReceived(const sp<AMessage> &msg) {switch (msg->what()) {case kWhatSourceNotify:{int32_t sourceIndex;CHECK(msg->findInt32("source-index", &sourceIndex));sp<SourceInfo> source = mSources.editItemAt(sourceIndex);int32_t what;CHECK(msg->findInt32("what", &what));if (what == SourceInfo::kNotifyReachedEOS|| what == SourceInfo::kNotifyStartFailed) {source->setEOSReceived();sp<ABuffer> buffer = source->lastAccessUnit();source->setLastAccessUnit(NULL);if (buffer != NULL) {writeTS();writeAccessUnit(sourceIndex, buffer);}++mNumSourcesDone;} else if (what == SourceInfo::kNotifyBuffer) {sp<ABuffer> buffer;CHECK(msg->findBuffer("buffer", &buffer));CHECK(source->lastAccessUnit() == NULL);int32_t oob;if (msg->findInt32("oob", &oob) && oob) {// This is codec specific data delivered out of band.// It can be written out immediately.writeTS();writeAccessUnit(sourceIndex, buffer);break;}// We don't just write out data as we receive it from// the various sources. That would essentially write them// out in random order (as the thread scheduler determines// how the messages are dispatched).// Instead we gather an access unit for all tracks and// write out the one with the smallest timestamp, then// request more data for the written out track.// Rinse, repeat.// If we don't have data on any track we don't write// anything just yet.source->setLastAccessUnit(buffer);ALOGV("lastAccessUnitTimeUs[%d] = %.2f secs",sourceIndex, source->lastAccessUnitTimeUs() / 1E6);int64_t minTimeUs = -1;size_t minIndex = 0;for (size_t i = 0; i < mSources.size(); ++i) {const sp<SourceInfo> &source = mSources.editItemAt(i);if (source->eosReceived()) {continue;}int64_t timeUs = source->lastAccessUnitTimeUs();if (timeUs < 0) {minTimeUs = -1;break;} else if (minTimeUs < 0 || timeUs < minTimeUs) {minTimeUs = timeUs;minIndex = i;}}if (minTimeUs < 0) {ALOGV("not all tracks have valid data.");break;}ALOGV("writing access unit at time %.2f secs (index %zu)",minTimeUs / 1E6, minIndex);source = mSources.editItemAt(minIndex);buffer = source->lastAccessUnit();source->setLastAccessUnit(NULL);writeTS();writeAccessUnit(minIndex, buffer);source->readMore();}break;}default:TRESPASS();}}
在这个函数里，收到数据，并写入到文件的是“what == SourceInfo::kNotifyBuffer”这个条件下的代码段。注意在这个代码段里的那个for循环，我们丢帧就是在这里丢的。
这个for循环的作用是干什么呢？它的作用是，选取当前录制的视频的几个源中，时间戳最小的那一个源的数据，并将选取的源的数据写入文件。这么做的原因上面的注释写了，大意是，一个视频会有几个源，分属不同的线程。因为在不同的线程，所以调度时间有先后顺序，有时视频数据已经读取到了，但是cpu现在调度的是音频源，视频数据就要等音频源写完数据后，再去写视频源的数据，这样就会导致声音和视频有错位。比如播放的时候，声音说完了，对应的画面过了一秒才播出来。
google的这个解释，似乎说的通，似乎有那么一丝的道理。但是实际上，这段代码逻辑却是有混乱不堪。再举个例子，比如当前收到的是视频帧，视频帧的timeUS，也就是时间戳是112233。然后第一次执行完这个for循环后，minTimeUs会等于112233，i=1。然后因为还有音频，会第二次执行这个for循环。假设这时音频的时间戳是112232，它比视频的时间戳小，那么，minTimeUS就被改成了112232， minIndex=2。好了，执行完上面两次for循环后，会马上执行source = mSources.editItemAt(minIndex);，去取出音频的数据，写入文件，然后再紧接着调用source->readMore();去继续读取音频的内容。
不知道大家有没有注意到，本来这次发送SourceInfo::kNotifyBuffer这个整个的源是视频源，但是到最后，写入的数据却是音频源的。那么视频源的数据到哪里去了呢？没错，居然被直接丢弃掉了，丢弃掉了........
不知道写这段逻辑的人的脑子是怎么长的，总之，这里的逻辑是个很明显的错误。想要解决这个问题也很简单，把这个fro循环去掉，SourceInfo::kNotifyBuffer这个源是谁发过来的，就写谁的数据，不用去管时间戳。因为mpeg2ts流数据，每个pes数据包中，都包含了它的时间戳。具体的可以看下面的代码：
void MPEG2TSWriter::writeAccessUnit(int32_t sourceIndex, const sp<ABuffer> &accessUnit) {........int64_t timeUs;CHECK(accessUnit->meta()->findInt64("timeUs", &timeUs));uint32_t PTS = (timeUs * 9ll) / 100ll;........}
再者，在视频文件里，无论是哪种格式的，音频轨和视频轨都是分开存放的。接收存储数据时，只用管当前轨道的数据是按先后顺序存放的就可以。所以，根本不需要多此一举，在收到某个源的数据后，还要和其他源的数据比时间戳。修改后的代码如下：
void MPEG2TSWriter::onMessageReceived(const sp<AMessage> &msg) {switch (msg->what()) {case kWhatSourceNotify:{int32_t sourceIndex;CHECK(msg->findInt32("source-index", &sourceIndex));sp<SourceInfo> source = mSources.editItemAt(sourceIndex);int32_t what;CHECK(msg->findInt32("what", &what));if (what == SourceInfo::kNotifyReachedEOS|| what == SourceInfo::kNotifyStartFailed) {source->setEOSReceived();sp<ABuffer> buffer = source->lastAccessUnit();source->setLastAccessUnit(NULL);if (buffer != NULL) {writeTS();writeAccessUnit(sourceIndex, buffer);}++mNumSourcesDone;} else if (what == SourceInfo::kNotifyBuffer) {sp<ABuffer> buffer;CHECK(msg->findBuffer("buffer", &buffer));CHECK(source->lastAccessUnit() == NULL);int32_t oob;if (msg->findInt32("oob", &oob) && oob) {// This is codec specific data delivered out of band.// It can be written out immediately.writeTS();writeAccessUnit(sourceIndex, buffer);break;}// We don't just write out data as we receive it from// the various sources. That would essentially write them// out in random order (as the thread scheduler determines// how the messages are dispatched).// Instead we gather an access unit for all tracks and// write out the one with the smallest timestamp, then// request more data for the written out track.// Rinse, repeat.// If we don't have data on any track we don't write// anything just yet.source->setLastAccessUnit(buffer);#if 0ALOGV("lastAccessUnitTimeUs[%d] = %.2f secs",sourceIndex, source->lastAccessUnitTimeUs() / 1E6);int64_t minTimeUs = -1;size_t minIndex = 0;for (size_t i = 0; i < mSources.size(); ++i) {const sp<SourceInfo> &source = mSources.editItemAt(i);if (source->eosReceived()) {continue;}int64_t timeUs = source->lastAccessUnitTimeUs();if (timeUs < 0) {minTimeUs = -1;break;} else if (minTimeUs < 0 || timeUs < minTimeUs) {minTimeUs = timeUs;minIndex = i;}}if (minTimeUs < 0) {ALOGV("not all tracks have valid data.");break;}ALOGV("writing access unit at time %.2f secs (index %zu)",minTimeUs / 1E6, minIndex);source = mSources.editItemAt(minIndex);#endifbuffer = source->lastAccessUnit();source->setLastAccessUnit(NULL);writeTS();//writeAccessUnit(minIndex, buffer);writeAccessUnit(sourceIndex, buffer); source->readMore();}break;}default:TRESPASS();}}
好了，上面这样修改后，经过反复测试验证，录下来的视频不存在丢帧的问题，丢帧的问题完美的解决了。
现在再来说说，怎么去提供mpeg2ts流的mediamuxer给java层使用。先上一段java上的测试代码：
MediaExtractor extractor;int trackCount;MediaMuxer muxer;HashMap<Integer, Integer> indexMap;private void cloneMediaUsingMuxer(FileDescriptor srcMedia, String dstMediaPath,int expectedTrackCount, int degrees, int fmt) throws IOException {// Set up MediaExtractor to read from the source.extractor = new MediaExtractor();extractor.setDataSource(srcMedia, 0, testFileLength);trackCount = extractor.getTrackCount();muxer = new MediaMuxer(dstMediaPath, fmt);indexMap = new HashMap<Integer, Integer>(trackCount);for (int i = 0; i < trackCount; i++) {extractor.selectTrack(i);MediaFormat format = extractor.getTrackFormat(i);int dstIndex = muxer.addTrack(format);indexMap.put(i, dstIndex);}if (degrees >= 0) {muxer.setOrientationHint(degrees);}muxer.start();Handler handler = new Handler();handler.postDelayed(new Runnable() {@Overridepublic void run() {// Copy the samples from MediaExtractor to MediaMuxer.boolean sawEOS = false;int bufferSize = MAX_SAMPLE_SIZE;int frameCount = 0;int offset = 100;ByteBuffer dstBuf = ByteBuffer.allocate(bufferSize);BufferInfo bufferInfo = new BufferInfo();while (!sawEOS) {bufferInfo.offset = offset;bufferInfo.size = extractor.readSampleData(dstBuf, offset);if (bufferInfo.size < 0) {sawEOS = true;bufferInfo.size = 0;} else {bufferInfo.presentationTimeUs = extractor.getSampleTime();bufferInfo.flags = extractor.getSampleFlags();int trackIndex = extractor.getSampleTrackIndex();muxer.writeSampleData(indexMap.get(trackIndex), dstBuf,bufferInfo);extractor.advance();frameCount++;}}muxer.stop();muxer.release();}//这里延时10毫秒执行，是因为mpeg2ts的muxer有时启动稍慢。如果writeSampleData的//的时候，muxer还没启动，就会报错}, 10);return;}
这段代码中，就做了一件事，那就是从给定的文件里，用MediaExtractor去抽出每一帧，然后再用MediaMuxer将抽出的帧，打包成指定格式的视频文件。我们的目的是将一个给定的视频，通过mediaMuxer打包成mpeg2ts流视频。但是从frameworks\base\media\java\android\media\MediaMuxer.java的setUpMediaMuxer里可以看出，目前android不支持转成mpeg2ts流。要想达到我们的目的，首先需要在setUpMediaMuxer这个函数里，将mpeg2ts格式给加上去。
public static final class OutputFormat {/* Do not change these values without updating their counterparts* in include/media/stagefright/MediaMuxer.h!*/private OutputFormat() {}/** MPEG4 media file format*/public static final int MUXER_OUTPUT_MPEG_4 = 0;/** WEBM media file format*/public static final int MUXER_OUTPUT_WEBM = 1;/** 3GPP media file format*/public static final int MUXER_OUTPUT_3GPP = 2;public static final int MUXER_OUTPUT_MPEG2TS = 3;};private void setUpMediaMuxer(@NonNull FileDescriptor fd, @Format int format) throws IOException {if (format != OutputFormat.MUXER_OUTPUT_MPEG_4 && format != OutputFormat.MUXER_OUTPUT_WEBM&& format != OutputFormat.MUXER_OUTPUT_3GPP && format != OutputFormat.MUXER_OUTPUT_MPEG2TS) {throw new IllegalArgumentException("format: " + format + " is invalid");}mNativeObject = nativeSetup(fd, format);mState = MUXER_STATE_INITIALIZED;mCloseGuard.open("release");}
在上面，我们新增了一个格式MUXER_OUTPUT_MPEG2TS 。然后这里就一步步的调到了frameworks\av\media\libstagefright\MediaMuxer.cpp，同样，我们需要在这个文件里，增加我们的格式：
enum OutputFormat {OUTPUT_FORMAT_MPEG_4= 0,OUTPUT_FORMAT_WEBM = 1,OUTPUT_FORMAT_THREE_GPP = 2, //add by mpeg2tsOUTPUT_FORMAT_YUNOVO_MPEG2TS= 3,OUTPUT_FORMAT_LIST_END // must be last - used to validate format type};MediaMuxer::MediaMuxer(int fd, OutputFormat format): mFormat(format),mState(UNINITIALIZED) {ALOGV("MediaMuxer start, format=%d", format); if (format == OUTPUT_FORMAT_MPEG_4 || format == OUTPUT_FORMAT_THREE_GPP) {mWriter = new MPEG4Writer(fd);} else if (format == OUTPUT_FORMAT_WEBM) {mWriter = new WebmWriter(fd);}//add mpeg2tselse if (format == OUTPUT_FORMAT_YUNOVO_MPEG2TS){mWriter = new MPEG2TSWriter(fd);}//add endif (mWriter != NULL) {mFileMeta = new MetaData;mState = INITIALIZED;}}
好了，到这里为止，从java到c++层的接口，就算是打通了。现在就可以使用extractor.readSampleData去抽取视频帧数据，然后使用muxer.writeSampleData去写mpeg2ts流文件了。
下面顺便说一下，这个抽帧和写帧的流程。我们在MediaMuxer.cpp里构建好Muxer后，就可以在java层上通过muxer.addTrack(format)，将源文件里的视频track和音频track甚至字幕track添加进来了。
ssize_t MediaMuxer::addTrack(const sp<AMessage> &format) {Mutex::Autolock autoLock(mMuxerLock);if (format.get() == NULL) {ALOGE("addTrack() get a null format");return -EINVAL;}if (mState != INITIALIZED) {ALOGE("addTrack() must be called after constructor and before start().");return INVALID_OPERATION;}sp<MetaData> trackMeta = new MetaData;convertMessageToMetaData(format, trackMeta);sp<MediaAdapter> newTrack = new MediaAdapter(trackMeta);status_t result = mWriter->addSource(newTrack); if (result == OK) { return mTrackList.add(newTrack); }return -1;}
我们注意到，这里的track，是一个MediaAdapter类。请大家记住这个类，因为后面我们在java层调用writeSampleData去写帧数据时，最终都是通过这个类去push buffer的。
status_t MediaMuxer::writeSampleData(const sp<ABuffer> &buffer, size_t trackIndex,int64_t timeUs, uint32_t flags) {Mutex::Autolock autoLock(mMuxerLock);ALOGV("MediaMuxer::writeSampleData trackIndex= %zu; timeUs= %" PRIu64, trackIndex, timeUs);if (buffer.get() == NULL) {ALOGE("WriteSampleData() get an NULL buffer.");return -EINVAL;}if (mState != STARTED) {ALOGE("WriteSampleData() is called in invalid state %d", mState);return INVALID_OPERATION;}if (trackIndex >= mTrackList.size()) {ALOGE("WriteSampleData() get an invalid index %zu", trackIndex);return -EINVAL;}ALOGV("MediaMuxer::writeSampleData buffer offset = %zu, length = %zu", buffer->offset(), buffer->size());MediaBuffer* mediaBuffer = new MediaBuffer(buffer);mediaBuffer->add_ref(); // Released in MediaAdapter::signalBufferReturned().mediaBuffer->set_range(buffer->offset(), buffer->size());sp<MetaData> sampleMetaData = mediaBuffer->meta_data();sampleMetaData->setInt64(kKeyTime, timeUs);// Just set the kKeyDecodingTime as the presentation time for now.sampleMetaData->setInt64(kKeyDecodingTime, timeUs);if (flags & MediaCodec::BUFFER_FLAG_SYNCFRAME) {sampleMetaData->setInt32(kKeyIsSyncFrame, true);}sp<MediaAdapter> currentTrack = mTrackList[trackIndex];// This pushBuffer will wait until the mediaBuffer is consumed.return currentTrack->pushBuffer(mediaBuffer);}
每写一帧时，都会在mediaMuxer.cpp里，调用MediaAdapter的接口，去pushBuffer。这个pushBuffer，将数据push到哪里去了，可以跟到frameworks\av\media\libstagefright\MediaAdapter.cpp里来看看：
void MediaAdapter::signalBufferReturned(MediaBuffer *buffer) {Mutex::Autolock autoLock(mAdapterLock);CHECK(buffer != NULL);buffer->setObserver(0);buffer->release();ALOGV("buffer returned %p", buffer);mBufferReturnedCond.signal();}status_t MediaAdapter::read(MediaBuffer **buffer, const ReadOptions * /* options */) {Mutex::Autolock autoLock(mAdapterLock);if (!mStarted) {ALOGV("Read before even started!");return ERROR_END_OF_STREAM;}while (mCurrentMediaBuffer == NULL && mStarted) {ALOGV("waiting @ read()");mBufferReadCond.wait(mAdapterLock);}if (!mStarted) {ALOGV("read interrupted after stop");CHECK(mCurrentMediaBuffer == NULL);return ERROR_END_OF_STREAM;}CHECK(mCurrentMediaBuffer != NULL);*buffer = mCurrentMediaBuffer;mCurrentMediaBuffer = NULL;(*buffer)->setObserver(this);return OK;}status_t MediaAdapter::pushBuffer(MediaBuffer *buffer) {if (buffer == NULL) {ALOGE("pushBuffer get an NULL buffer");return -EINVAL;}Mutex::Autolock autoLock(mAdapterLock);if (!mStarted) {ALOGE("pushBuffer called before start");return INVALID_OPERATION;}mCurrentMediaBuffer = buffer;mBufferReadCond.signal();ALOGV("wait for the buffer returned @ pushBuffer! %p", buffer);mBufferReturnedCond.wait(mAdapterLock);return OK;}
从pushBuffer函数里可以看到，每当mCurrentMediaBuffer = buffer;这样赋值后，就会通过mBufferReadCond.signal();发送信号。这个mBufferReadCond的接收者在read函数里。当read收到消息后，就会将值通过read的指针传送到调用read的地方。调用read的地方是frameworks\av\media\libstagefright\MPEG2TSWriter.cpp里的下面的函数：
void MPEG2TSWriter::SourceInfo::onMessageReceived(const sp<AMessage> &msg) {switch (msg->what()) {......case kWhatRead:{MediaBuffer *buffer;status_t err = mSource->read(&buffer);if (err != OK && err != INFO_FORMAT_CHANGED) {sp<AMessage> notify = mNotify->dup();notify->setInt32("what", kNotifyReachedEOS);notify->setInt32("status", err);notify->post();break;}if (err == OK) {if (mStreamType == 0x0f && mAACCodecSpecificData == NULL) {// The first audio buffer must contain CSD if not received yet.CHECK_GE(buffer->range_length(), 2u);mAACCodecSpecificData = new ABuffer(buffer->range_length());memcpy(mAACCodecSpecificData->data(),(const uint8_t *)buffer->data()+ buffer->range_offset(),buffer->range_length());readMore();} else if (buffer->range_length() > 0) {if (mStreamType == 0x0f) {appendAACFrames(buffer);} else {appendAVCFrame(buffer);}} else {readMore();}buffer->release();buffer = NULL;}// Do not read more data until told to.break;}default:TRESPASS();}}
这里在读到数据后，通过判断是音频的还是视频的，丢给不同的函数去处理。比如是视频的话，就会丢给appendAVCFrame去处理。
void MPEG2TSWriter::SourceInfo::appendAVCFrame(MediaBuffer *buffer) {sp<AMessage> notify = mNotify->dup();notify->setInt32("what", kNotifyBuffer);if (mBuffer == NULL || buffer->range_length() > mBuffer->capacity()) {mBuffer = new ABuffer(buffer->range_length());}mBuffer->setRange(0, 0);memcpy(mBuffer->data(),(const uint8_t *)buffer->data()+ buffer->range_offset(),buffer->range_length());int64_t timeUs;CHECK(buffer->meta_data()->findInt64(kKeyTime, &timeUs));mBuffer->meta()->setInt64("timeUs", timeUs);int32_t isSync;if (buffer->meta_data()->findInt32(kKeyIsSyncFrame, &isSync)&& isSync != 0) {mBuffer->meta()->setInt32("isSync", true);}mBuffer->setRange(0, buffer->range_length());notify->setBuffer("buffer", mBuffer);notify->post();}
从这个函数里我们可以看到，appendAVCFrame函数，只对数据帧设置时间戳和同步标志后，就通过一个通知，丢给了MPEG2TSWriter::onMessageReceived去处理。MPEG2TSWriter::onMessageReceived收到帧后的处理过程，就是最开始咱们讨论的那个地方了。
另外，如果我们是从指定的mpeg2ts流文件里抽帧，然后再通过mpeg2tswriter去打包成一个新的ts流的话，有一个地方需要注意。那就是MPEG2TSWriter::SourceInfo::appendAACFrames(MediaBuffer *buffer)这个函数里的开始的地方，加个判断：
if(mIsMuxer){buffer->set_range(7, buffer->range_length()-7);}
因为这里加个属性来判断,当是在muxer时,就要加上下面这一行.因为现有的ts视频,每一帧音频已经加上了
7个字节的音频头.如果不将这7个字节的音频头给去掉,会导致每一帧音频上又多加了一个7字节的音频头.
这样的后果会导致大部份的播放器识别不了这个音频,播放不出了声音.
到此为止，我们就将mpeg2ts的流程梳理完成了，并且修正了录相丢帧的bug，封装了mpeg2ts muxer java层接口。
如果觉得《android mpeg2ts 流媒体打包MediaMuxer 和 录制MPEG2TSWriter 以及抽帧MPEG2TSExtractor》对你有帮助，请点赞、收藏，并留下你的观点哦！
